2023-04-07 09:53:32,511 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1 2023-04-07 09:53:32,525 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-07 09:53:32,542 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-07 09:53:32,543 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/cluster_33ba0ff8-c2ef-b3a6-4e4d-4cc1067ceb2e, deleteOnExit=true 2023-04-07 09:53:32,543 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-07 09:53:32,544 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/test.cache.data in system properties and HBase conf 2023-04-07 09:53:32,544 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/hadoop.tmp.dir in system properties and HBase conf 2023-04-07 09:53:32,545 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/hadoop.log.dir in system properties and HBase conf 2023-04-07 09:53:32,545 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-07 09:53:32,546 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-07 09:53:32,546 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-07 09:53:32,662 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-07 09:53:33,085 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-07 09:53:33,090 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-07 09:53:33,091 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-07 09:53:33,091 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-07 09:53:33,092 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-07 09:53:33,092 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-07 09:53:33,093 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-07 09:53:33,093 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-07 09:53:33,094 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-07 09:53:33,094 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-07 09:53:33,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/nfs.dump.dir in system properties and HBase conf 2023-04-07 09:53:33,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/java.io.tmpdir in system properties and HBase conf 2023-04-07 09:53:33,095 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-07 09:53:33,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-07 09:53:33,096 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-07 09:53:33,697 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-07 09:53:33,701 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-07 09:53:34,008 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-07 09:53:34,200 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-07 09:53:34,220 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-07 09:53:34,265 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-07 09:53:34,296 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/java.io.tmpdir/Jetty_localhost_36507_hdfs____.bhv82l/webapp 2023-04-07 09:53:34,436 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36507 2023-04-07 09:53:34,479 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-07 09:53:34,480 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-07 09:53:34,909 WARN [Listener at localhost/36015] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-07 09:53:35,062 WARN [Listener at localhost/36015] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-07 09:53:35,086 WARN [Listener at localhost/36015] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-07 09:53:35,092 INFO [Listener at localhost/36015] log.Slf4jLog(67): jetty-6.1.26 2023-04-07 09:53:35,097 INFO [Listener at localhost/36015] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/java.io.tmpdir/Jetty_localhost_38331_datanode____.vxde4t/webapp 2023-04-07 09:53:35,200 INFO [Listener at localhost/36015] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38331 2023-04-07 09:53:35,585 WARN [Listener at localhost/39511] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-07 09:53:36,117 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x37bbddf4e2969701: Processing first storage report for DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b from datanode 70ccd2b6-8a23-49ed-96bf-90cb8121a8b4 2023-04-07 09:53:36,119 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x37bbddf4e2969701: from storage DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b node DatanodeRegistration(127.0.0.1:34041, datanodeUuid=70ccd2b6-8a23-49ed-96bf-90cb8121a8b4, infoPort=45557, infoSecurePort=0, ipcPort=39511, storageInfo=lv=-57;cid=testClusterID;nsid=1412716843;c=1680861213773), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-07 09:53:36,119 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x37bbddf4e2969701: Processing first storage report for DS-fd4ab1c2-98f4-447b-98c8-316ef7f895eb from datanode 70ccd2b6-8a23-49ed-96bf-90cb8121a8b4 2023-04-07 09:53:36,119 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x37bbddf4e2969701: from storage DS-fd4ab1c2-98f4-447b-98c8-316ef7f895eb node DatanodeRegistration(127.0.0.1:34041, datanodeUuid=70ccd2b6-8a23-49ed-96bf-90cb8121a8b4, infoPort=45557, infoSecurePort=0, ipcPort=39511, storageInfo=lv=-57;cid=testClusterID;nsid=1412716843;c=1680861213773), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-07 09:53:36,131 DEBUG [Listener at localhost/39511] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1 2023-04-07 09:53:36,193 INFO [Listener at localhost/39511] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/cluster_33ba0ff8-c2ef-b3a6-4e4d-4cc1067ceb2e/zookeeper_0, clientPort=49660, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/cluster_33ba0ff8-c2ef-b3a6-4e4d-4cc1067ceb2e/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/cluster_33ba0ff8-c2ef-b3a6-4e4d-4cc1067ceb2e/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-07 09:53:36,207 INFO [Listener at localhost/39511] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=49660 2023-04-07 09:53:36,216 INFO [Listener at localhost/39511] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 09:53:36,219 INFO [Listener at localhost/39511] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 09:53:36,883 INFO [Listener at localhost/39511] util.FSUtils(471): Created version file at hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e with version=8 2023-04-07 09:53:36,884 INFO [Listener at localhost/39511] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/hbase-staging 2023-04-07 09:53:37,203 INFO [Listener at localhost/39511] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-07 09:53:37,719 INFO [Listener at localhost/39511] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-07 09:53:37,755 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 09:53:37,756 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-07 09:53:37,756 INFO [Listener at localhost/39511] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-07 09:53:37,756 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 09:53:37,757 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-07 09:53:37,912 INFO [Listener at localhost/39511] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-07 09:53:37,992 DEBUG [Listener at localhost/39511] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-07 09:53:38,090 INFO [Listener at localhost/39511] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46673 2023-04-07 09:53:38,101 INFO [Listener at localhost/39511] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 09:53:38,103 INFO [Listener at localhost/39511] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 09:53:38,124 INFO [Listener at localhost/39511] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46673 connecting to ZooKeeper ensemble=127.0.0.1:49660 2023-04-07 09:53:38,164 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:466730x0, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-07 09:53:38,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46673-0x1015e0aa7780000 connected 2023-04-07 09:53:38,197 DEBUG [Listener at localhost/39511] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-07 09:53:38,198 DEBUG [Listener at localhost/39511] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 09:53:38,201 DEBUG [Listener at localhost/39511] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-07 09:53:38,209 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46673 2023-04-07 09:53:38,210 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46673 2023-04-07 09:53:38,210 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46673 2023-04-07 09:53:38,212 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46673 2023-04-07 09:53:38,212 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46673 2023-04-07 09:53:38,219 INFO [Listener at localhost/39511] master.HMaster(444): hbase.rootdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e, hbase.cluster.distributed=false 2023-04-07 09:53:38,289 INFO [Listener at localhost/39511] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-07 09:53:38,289 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 09:53:38,290 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-07 09:53:38,290 INFO [Listener at localhost/39511] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-07 09:53:38,290 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-07 09:53:38,290 INFO [Listener at localhost/39511] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-07 09:53:38,295 INFO [Listener at localhost/39511] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-07 09:53:38,298 INFO [Listener at localhost/39511] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35881 2023-04-07 09:53:38,300 INFO [Listener at localhost/39511] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-07 09:53:38,307 DEBUG [Listener at localhost/39511] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-07 09:53:38,307 INFO [Listener at localhost/39511] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 09:53:38,310 INFO [Listener at localhost/39511] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 09:53:38,312 INFO [Listener at localhost/39511] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35881 connecting to ZooKeeper ensemble=127.0.0.1:49660 2023-04-07 09:53:38,317 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:358810x0, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-07 09:53:38,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35881-0x1015e0aa7780001 connected 2023-04-07 09:53:38,318 DEBUG [Listener at localhost/39511] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-07 09:53:38,319 DEBUG [Listener at localhost/39511] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 09:53:38,320 DEBUG [Listener at localhost/39511] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-07 09:53:38,321 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35881 2023-04-07 09:53:38,321 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35881 2023-04-07 09:53:38,322 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35881 2023-04-07 09:53:38,322 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35881 2023-04-07 09:53:38,322 DEBUG [Listener at localhost/39511] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35881 2023-04-07 09:53:38,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:38,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-07 09:53:38,334 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:38,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-07 09:53:38,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-07 09:53:38,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:38,354 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-07 09:53:38,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46673,1680861217027 from backup master directory 2023-04-07 09:53:38,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-07 09:53:38,359 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:38,359 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-07 09:53:38,359 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-07 09:53:38,359 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:38,362 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-07 09:53:38,363 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-07 09:53:38,856 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/hbase.id with ID: 88ff4bad-32ab-4c19-9339-bf7e7aaaf53d 2023-04-07 09:53:38,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-07 09:53:38,927 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:38,973 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x08220e1e to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:39,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cc6c4c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:39,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:53:39,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-07 09:53:39,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-07 09:53:39,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-07 09:53:39,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:53:39,071 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:53:39,073 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-07 09:53:39,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store-tmp 2023-04-07 09:53:39,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:39,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-07 09:53:39,542 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:53:39,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:53:39,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-07 09:53:39,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:53:39,543 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:53:39,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-07 09:53:39,545 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/WALs/jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:39,567 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46673%2C1680861217027, suffix=, logDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/WALs/jenkins-hbase4.apache.org,46673,1680861217027, archiveDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/oldWALs, maxLogs=10 2023-04-07 09:53:39,619 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34041,DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b,DISK] 2023-04-07 09:53:39,627 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:53:39,685 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/WALs/jenkins-hbase4.apache.org,46673,1680861217027/jenkins-hbase4.apache.org%2C46673%2C1680861217027.1680861219578 2023-04-07 09:53:39,686 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34041,DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b,DISK]] 2023-04-07 09:53:39,687 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:53:39,687 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:39,690 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-07 09:53:39,691 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-07 09:53:39,749 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-07 09:53:39,756 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-07 09:53:39,779 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-07 09:53:39,790 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:39,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-07 09:53:39,797 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-07 09:53:39,812 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-07 09:53:39,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:53:39,817 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64673288, jitterRate=-0.0362929105758667}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-07 09:53:39,818 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-07 09:53:39,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-07 09:53:39,839 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-07 09:53:39,839 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-07 09:53:39,841 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-07 09:53:39,843 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-07 09:53:39,876 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-04-07 09:53:39,876 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-07 09:53:39,901 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-07 09:53:39,906 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-07 09:53:39,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-07 09:53:39,933 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-07 09:53:39,935 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-07 09:53:39,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-07 09:53:39,944 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-07 09:53:39,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:39,948 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-07 09:53:39,949 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-07 09:53:39,960 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-07 09:53:39,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-07 09:53:39,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-07 09:53:39,965 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:39,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46673,1680861217027, sessionid=0x1015e0aa7780000, setting cluster-up flag (Was=false) 2023-04-07 09:53:39,980 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:39,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-07 09:53:39,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:39,992 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:39,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-07 09:53:39,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:40,000 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.hbase-snapshot/.tmp 2023-04-07 09:53:40,027 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(952): ClusterId : 88ff4bad-32ab-4c19-9339-bf7e7aaaf53d 2023-04-07 09:53:40,031 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-07 09:53:40,037 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-07 09:53:40,037 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-07 09:53:40,040 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-07 09:53:40,041 DEBUG [RS:0;jenkins-hbase4:35881] zookeeper.ReadOnlyZKClient(139): Connect 0x49037e10 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:40,047 DEBUG [RS:0;jenkins-hbase4:35881] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47c49940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:40,047 DEBUG [RS:0;jenkins-hbase4:35881] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43fb1560, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-07 09:53:40,071 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35881 2023-04-07 09:53:40,075 INFO [RS:0;jenkins-hbase4:35881] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-07 09:53:40,075 INFO [RS:0;jenkins-hbase4:35881] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-07 09:53:40,075 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1023): About to register with Master. 2023-04-07 09:53:40,078 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46673,1680861217027 with isa=jenkins-hbase4.apache.org/172.31.14.131:35881, startcode=1680861218288 2023-04-07 09:53:40,093 DEBUG [RS:0;jenkins-hbase4:35881] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-07 09:53:40,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-07 09:53:40,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 09:53:40,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 09:53:40,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 09:53:40,142 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-07 09:53:40,142 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-07 09:53:40,143 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,143 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-07 09:53:40,143 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680861250144 2023-04-07 09:53:40,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-07 09:53:40,150 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-07 09:53:40,151 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-07 09:53:40,152 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49713, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-07 09:53:40,157 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-07 09:53:40,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-07 09:53:40,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:40,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-07 09:53:40,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-07 09:53:40,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-07 09:53:40,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-07 09:53:40,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-07 09:53:40,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-07 09:53:40,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-07 09:53:40,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-07 09:53:40,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-07 09:53:40,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680861220179,5,FailOnTimeoutGroup] 2023-04-07 09:53:40,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680861220180,5,FailOnTimeoutGroup] 2023-04-07 09:53:40,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-07 09:53:40,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,195 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(2829): Master is not running yet 2023-04-07 09:53:40,195 WARN [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-07 09:53:40,296 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46673,1680861217027 with isa=jenkins-hbase4.apache.org/172.31.14.131:35881, startcode=1680861218288 2023-04-07 09:53:40,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:40,308 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e 2023-04-07 09:53:40,308 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36015 2023-04-07 09:53:40,308 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-07 09:53:40,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-07 09:53:40,314 DEBUG [RS:0;jenkins-hbase4:35881] zookeeper.ZKUtil(162): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:40,314 WARN [RS:0;jenkins-hbase4:35881] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-07 09:53:40,315 INFO [RS:0;jenkins-hbase4:35881] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-07 09:53:40,315 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/WALs/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:40,316 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35881,1680861218288] 2023-04-07 09:53:40,323 DEBUG [RS:0;jenkins-hbase4:35881] zookeeper.ZKUtil(162): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:40,333 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-07 09:53:40,341 INFO [RS:0;jenkins-hbase4:35881] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-07 09:53:40,360 INFO [RS:0;jenkins-hbase4:35881] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-07 09:53:40,365 INFO [RS:0;jenkins-hbase4:35881] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-07 09:53:40,365 INFO [RS:0;jenkins-hbase4:35881] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,366 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-07 09:53:40,374 INFO [RS:0;jenkins-hbase4:35881] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,374 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,375 DEBUG [RS:0;jenkins-hbase4:35881] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-07 09:53:40,376 INFO [RS:0;jenkins-hbase4:35881] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,376 INFO [RS:0;jenkins-hbase4:35881] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,377 INFO [RS:0;jenkins-hbase4:35881] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,393 INFO [RS:0;jenkins-hbase4:35881] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-07 09:53:40,395 INFO [RS:0;jenkins-hbase4:35881] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35881,1680861218288-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:40,410 INFO [RS:0;jenkins-hbase4:35881] regionserver.Replication(203): jenkins-hbase4.apache.org,35881,1680861218288 started 2023-04-07 09:53:40,411 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35881,1680861218288, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35881, sessionid=0x1015e0aa7780001 2023-04-07 09:53:40,411 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-07 09:53:40,411 DEBUG [RS:0;jenkins-hbase4:35881] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:40,411 DEBUG [RS:0;jenkins-hbase4:35881] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35881,1680861218288' 2023-04-07 09:53:40,411 DEBUG [RS:0;jenkins-hbase4:35881] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:40,412 DEBUG [RS:0;jenkins-hbase4:35881] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:40,412 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-07 09:53:40,412 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-07 09:53:40,413 DEBUG [RS:0;jenkins-hbase4:35881] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:40,413 DEBUG [RS:0;jenkins-hbase4:35881] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35881,1680861218288' 2023-04-07 09:53:40,413 DEBUG [RS:0;jenkins-hbase4:35881] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-07 09:53:40,413 DEBUG [RS:0;jenkins-hbase4:35881] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-07 09:53:40,414 DEBUG [RS:0;jenkins-hbase4:35881] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-07 09:53:40,414 INFO [RS:0;jenkins-hbase4:35881] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-07 09:53:40,414 INFO [RS:0;jenkins-hbase4:35881] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-07 09:53:40,525 INFO [RS:0;jenkins-hbase4:35881] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35881%2C1680861218288, suffix=, logDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/WALs/jenkins-hbase4.apache.org,35881,1680861218288, archiveDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/oldWALs, maxLogs=32 2023-04-07 09:53:40,543 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34041,DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b,DISK] 2023-04-07 09:53:40,547 INFO [RS:0;jenkins-hbase4:35881] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/WALs/jenkins-hbase4.apache.org,35881,1680861218288/jenkins-hbase4.apache.org%2C35881%2C1680861218288.1680861220527 2023-04-07 09:53:40,547 DEBUG [RS:0;jenkins-hbase4:35881] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34041,DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b,DISK]] 2023-04-07 09:53:40,593 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-07 09:53:40,594 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-07 09:53:40,594 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e 2023-04-07 09:53:41,008 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:41,011 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-07 09:53:41,013 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/info 2023-04-07 09:53:41,014 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-07 09:53:41,015 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:41,016 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-07 09:53:41,018 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/rep_barrier 2023-04-07 09:53:41,019 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-07 09:53:41,020 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:41,020 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-07 09:53:41,022 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/table 2023-04-07 09:53:41,022 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-07 09:53:41,023 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:41,025 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740 2023-04-07 09:53:41,026 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740 2023-04-07 09:53:41,029 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:53:41,031 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-07 09:53:41,034 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:53:41,035 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72071865, jitterRate=0.07395447790622711}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:53:41,035 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-07 09:53:41,036 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-07 09:53:41,036 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-07 09:53:41,036 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-07 09:53:41,036 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-07 09:53:41,036 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-07 09:53:41,037 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-07 09:53:41,037 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-07 09:53:41,041 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-07 09:53:41,041 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-07 09:53:41,049 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-07 09:53:41,061 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-07 09:53:41,063 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-07 09:53:41,215 DEBUG [jenkins-hbase4:46673] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-07 09:53:41,218 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35881,1680861218288, state=OPENING 2023-04-07 09:53:41,225 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-07 09:53:41,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:41,227 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-07 09:53:41,229 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:53:41,417 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:41,420 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-07 09:53:41,424 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46698, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-07 09:53:41,437 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-07 09:53:41,437 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-07 09:53:41,441 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35881%2C1680861218288.meta, suffix=.meta, logDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/WALs/jenkins-hbase4.apache.org,35881,1680861218288, archiveDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/oldWALs, maxLogs=32 2023-04-07 09:53:41,459 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34041,DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b,DISK] 2023-04-07 09:53:41,463 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/WALs/jenkins-hbase4.apache.org,35881,1680861218288/jenkins-hbase4.apache.org%2C35881%2C1680861218288.meta.1680861221442.meta 2023-04-07 09:53:41,464 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34041,DS-ea5dffec-63c7-4816-8c8a-5aa1e345834b,DISK]] 2023-04-07 09:53:41,464 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:53:41,466 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-07 09:53:41,482 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-07 09:53:41,488 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-07 09:53:41,496 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-07 09:53:41,496 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:41,496 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-07 09:53:41,497 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-07 09:53:41,500 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-07 09:53:41,501 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/info 2023-04-07 09:53:41,501 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/info 2023-04-07 09:53:41,502 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-07 09:53:41,503 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:41,503 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-07 09:53:41,504 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/rep_barrier 2023-04-07 09:53:41,504 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/rep_barrier 2023-04-07 09:53:41,505 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-07 09:53:41,505 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:41,506 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-07 09:53:41,507 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/table 2023-04-07 09:53:41,507 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/table 2023-04-07 09:53:41,507 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-07 09:53:41,508 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:41,510 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740 2023-04-07 09:53:41,512 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740 2023-04-07 09:53:41,516 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:53:41,519 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-07 09:53:41,521 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67254256, jitterRate=0.0021665096282958984}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:53:41,521 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-07 09:53:41,533 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680861221406 2023-04-07 09:53:41,550 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-07 09:53:41,550 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-07 09:53:41,551 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35881,1680861218288, state=OPEN 2023-04-07 09:53:41,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-07 09:53:41,556 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-07 09:53:41,560 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-07 09:53:41,560 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35881,1680861218288 in 327 msec 2023-04-07 09:53:41,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-07 09:53:41,565 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 512 msec 2023-04-07 09:53:41,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5200 sec 2023-04-07 09:53:41,569 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680861221569, completionTime=-1 2023-04-07 09:53:41,570 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-07 09:53:41,570 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-07 09:53:41,631 DEBUG [hconnection-0x2e4fa55b-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:41,635 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:41,656 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-07 09:53:41,656 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680861281656 2023-04-07 09:53:41,656 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680861341656 2023-04-07 09:53:41,656 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 86 msec 2023-04-07 09:53:41,679 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46673,1680861217027-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:41,680 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46673,1680861217027-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:41,680 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46673,1680861217027-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:41,681 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46673, period=300000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:41,682 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-07 09:53:41,687 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-07 09:53:41,695 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-07 09:53:41,696 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-07 09:53:41,708 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-07 09:53:41,710 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 09:53:41,713 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 09:53:41,736 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/hbase/namespace/18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:41,738 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/hbase/namespace/18a621e37938d05f473a21997b77c76c empty. 2023-04-07 09:53:41,739 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/hbase/namespace/18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:41,739 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-07 09:53:42,160 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-07 09:53:42,165 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 18a621e37938d05f473a21997b77c76c, NAME => 'hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp 2023-04-07 09:53:42,579 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:42,579 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 18a621e37938d05f473a21997b77c76c, disabling compactions & flushes 2023-04-07 09:53:42,579 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:53:42,579 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:53:42,579 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. after waiting 0 ms 2023-04-07 09:53:42,579 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:53:42,579 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:53:42,579 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 18a621e37938d05f473a21997b77c76c: 2023-04-07 09:53:42,584 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 09:53:42,600 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680861222587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861222587"}]},"ts":"1680861222587"} 2023-04-07 09:53:42,627 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 09:53:42,629 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 09:53:42,634 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861222629"}]},"ts":"1680861222629"} 2023-04-07 09:53:42,638 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-07 09:53:42,650 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=18a621e37938d05f473a21997b77c76c, ASSIGN}] 2023-04-07 09:53:42,652 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=18a621e37938d05f473a21997b77c76c, ASSIGN 2023-04-07 09:53:42,654 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=18a621e37938d05f473a21997b77c76c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=false 2023-04-07 09:53:42,805 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=18a621e37938d05f473a21997b77c76c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:42,806 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680861222805"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861222805"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861222805"}]},"ts":"1680861222805"} 2023-04-07 09:53:42,812 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 18a621e37938d05f473a21997b77c76c, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:53:42,971 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:53:42,972 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 18a621e37938d05f473a21997b77c76c, NAME => 'hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:53:42,973 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:42,973 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:42,973 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:42,973 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:42,975 INFO [StoreOpener-18a621e37938d05f473a21997b77c76c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:42,977 DEBUG [StoreOpener-18a621e37938d05f473a21997b77c76c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/info 2023-04-07 09:53:42,977 DEBUG [StoreOpener-18a621e37938d05f473a21997b77c76c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/info 2023-04-07 09:53:42,978 INFO [StoreOpener-18a621e37938d05f473a21997b77c76c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 18a621e37938d05f473a21997b77c76c columnFamilyName info 2023-04-07 09:53:42,979 INFO [StoreOpener-18a621e37938d05f473a21997b77c76c-1] regionserver.HStore(310): Store=18a621e37938d05f473a21997b77c76c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:42,980 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:42,981 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:42,986 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 18a621e37938d05f473a21997b77c76c 2023-04-07 09:53:42,990 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:53:42,991 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 18a621e37938d05f473a21997b77c76c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71901068, jitterRate=0.07140940427780151}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-07 09:53:42,991 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 18a621e37938d05f473a21997b77c76c: 2023-04-07 09:53:42,993 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c., pid=6, masterSystemTime=1680861222966 2023-04-07 09:53:42,998 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:53:42,998 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:53:42,999 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=18a621e37938d05f473a21997b77c76c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:43,000 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680861222999"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861222999"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861222999"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861222999"}]},"ts":"1680861222999"} 2023-04-07 09:53:43,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-07 09:53:43,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 18a621e37938d05f473a21997b77c76c, server=jenkins-hbase4.apache.org,35881,1680861218288 in 193 msec 2023-04-07 09:53:43,014 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-07 09:53:43,015 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=18a621e37938d05f473a21997b77c76c, ASSIGN in 360 msec 2023-04-07 09:53:43,016 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 09:53:43,016 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861223016"}]},"ts":"1680861223016"} 2023-04-07 09:53:43,021 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-07 09:53:43,025 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 09:53:43,028 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3280 sec 2023-04-07 09:53:43,111 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-07 09:53:43,113 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-07 09:53:43,113 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:53:43,154 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-07 09:53:43,170 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-07 09:53:43,175 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-07 09:53:43,187 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-07 09:53:43,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-07 09:53:43,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-07 09:53:43,212 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-07 09:53:43,215 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-07 09:53:43,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.855sec 2023-04-07 09:53:43,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-07 09:53:43,219 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-07 09:53:43,219 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-07 09:53:43,220 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46673,1680861217027-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-07 09:53:43,221 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46673,1680861217027-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-07 09:53:43,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-07 09:53:43,236 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:43,241 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:43,254 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:43,265 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:43,275 INFO [Listener at localhost/39511] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:53:43,295 INFO [Listener at localhost/39511] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=317, ProcessCount=169, AvailableMemoryMB=3609 2023-04-07 09:53:43,311 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 09:53:43,314 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 09:53:43,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:53:43,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:53:43,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 09:53:43,363 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 09:53:43,366 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 09:53:43,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-07 09:53:43,369 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:43,370 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b empty. 2023-04-07 09:53:43,372 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:43,372 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:53:43,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 09:53:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 09:53:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 09:53:43,794 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:53:43,796 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0b39eaaa997d54d62464302ce4e71f9b, NAME => 'TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp 2023-04-07 09:53:43,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 09:53:44,209 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:44,209 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0b39eaaa997d54d62464302ce4e71f9b, disabling compactions & flushes 2023-04-07 09:53:44,209 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,209 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,209 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. after waiting 0 ms 2023-04-07 09:53:44,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,210 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:44,214 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 09:53:44,215 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861224215"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861224215"}]},"ts":"1680861224215"} 2023-04-07 09:53:44,218 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 09:53:44,219 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 09:53:44,220 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861224220"}]},"ts":"1680861224220"} 2023-04-07 09:53:44,222 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 09:53:44,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b39eaaa997d54d62464302ce4e71f9b, ASSIGN}] 2023-04-07 09:53:44,228 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b39eaaa997d54d62464302ce4e71f9b, ASSIGN 2023-04-07 09:53:44,229 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b39eaaa997d54d62464302ce4e71f9b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=false 2023-04-07 09:53:44,380 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0b39eaaa997d54d62464302ce4e71f9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:44,380 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861224380"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861224380"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861224380"}]},"ts":"1680861224380"} 2023-04-07 09:53:44,384 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:53:44,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 09:53:44,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0b39eaaa997d54d62464302ce4e71f9b, NAME => 'TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:53:44,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:53:44,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,546 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,548 DEBUG [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A 2023-04-07 09:53:44,548 DEBUG [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A 2023-04-07 09:53:44,549 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:53:44,550 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b39eaaa997d54d62464302ce4e71f9b columnFamilyName A 2023-04-07 09:53:44,551 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.HStore(310): Store=0b39eaaa997d54d62464302ce4e71f9b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:44,551 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,553 DEBUG [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B 2023-04-07 09:53:44,553 DEBUG [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B 2023-04-07 09:53:44,553 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:53:44,553 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b39eaaa997d54d62464302ce4e71f9b columnFamilyName B 2023-04-07 09:53:44,554 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.HStore(310): Store=0b39eaaa997d54d62464302ce4e71f9b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:44,554 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,556 DEBUG [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C 2023-04-07 09:53:44,556 DEBUG [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C 2023-04-07 09:53:44,556 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:53:44,557 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b39eaaa997d54d62464302ce4e71f9b columnFamilyName C 2023-04-07 09:53:44,557 INFO [StoreOpener-0b39eaaa997d54d62464302ce4e71f9b-1] regionserver.HStore(310): Store=0b39eaaa997d54d62464302ce4e71f9b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:53:44,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:53:44,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:44,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:53:44,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0b39eaaa997d54d62464302ce4e71f9b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62702293, jitterRate=-0.06566302478313446}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:53:44,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:44,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., pid=11, masterSystemTime=1680861224537 2023-04-07 09:53:44,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:44,575 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0b39eaaa997d54d62464302ce4e71f9b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:44,575 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861224575"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861224575"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861224575"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861224575"}]},"ts":"1680861224575"} 2023-04-07 09:53:44,581 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-07 09:53:44,581 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 in 194 msec 2023-04-07 09:53:44,584 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-07 09:53:44,584 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b39eaaa997d54d62464302ce4e71f9b, ASSIGN in 355 msec 2023-04-07 09:53:44,585 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 09:53:44,586 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861224585"}]},"ts":"1680861224585"} 2023-04-07 09:53:44,588 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 09:53:44,592 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 09:53:44,594 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2340 sec 2023-04-07 09:53:45,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-07 09:53:45,493 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-07 09:53:45,498 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,503 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,506 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,509 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,512 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 09:53:45,515 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 09:53:45,524 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,528 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,529 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,533 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,533 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,538 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,538 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,542 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,543 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,546 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,547 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,551 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,551 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,580 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,581 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,585 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,585 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:53:45,589 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:53:45,595 DEBUG [hconnection-0x554a3b74-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,596 DEBUG [hconnection-0x690358fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,600 DEBUG [hconnection-0x15810e2e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,600 DEBUG [hconnection-0x55edf468-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,601 DEBUG [hconnection-0x5e67f105-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,601 DEBUG [hconnection-0x14f57e8a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,602 DEBUG [hconnection-0x72f88c55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,602 DEBUG [hconnection-0x74db5e90-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,603 DEBUG [hconnection-0x48d126f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:53:45,607 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,608 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,613 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,614 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,614 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,615 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,620 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,620 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,621 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:53:45,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:45,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:45,644 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:45,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:45,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:45,650 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:45,651 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:45,651 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:45,653 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:45,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:45,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:45,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:45,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:45,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:45,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:45,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:45,669 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:45,670 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:45,671 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:45,670 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:45,674 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:45,677 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-07 09:53:45,688 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:45,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:45,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:45,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:45,690 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:45,690 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:45,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:45,706 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:45,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861225794, max timestamp=1680861225824Num uniques 859; 2023-04-07 09:53:45,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861225794, max timestamp=1680861225857Num uniques 859; 2023-04-07 09:53:45,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:53:45,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861225805, max timestamp=1680861225824Num uniques 859; 2023-04-07 09:53:45,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:53:45,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:53:45,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:45,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861285927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861225922, max timestamp=1680861225930Num uniques 1068; 2023-04-07 09:53:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-07 09:53:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861225922, max timestamp=1680861225935Num uniques 1068; 2023-04-07 09:53:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861225922, max timestamp=1680861225930Num uniques 1009; 2023-04-07 09:53:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-07 09:53:45,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-07 09:53:45,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:45,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861285983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:45,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:45,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861285983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:45,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680861225965, max timestamp=1680861225981Num uniques 1009; 2023-04-07 09:53:45,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=14972, suffix heap size=132904,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=327 2023-04-07 09:53:46,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861225965, max timestamp=1680861225981Num uniques 1009; 2023-04-07 09:53:46,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-07 09:53:46,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861225965, max timestamp=1680861225981Num uniques 1009; 2023-04-07 09:53:46,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-07 09:53:46,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861286055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861286060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861226040, max timestamp=1680861226057Num uniques 1009; 2023-04-07 09:53:46,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861226040, max timestamp=1680861226057Num uniques 1009; 2023-04-07 09:53:46,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-07 09:53:46,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-07 09:53:46,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861286063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861226044, max timestamp=1680861226057Num uniques 1186; 2023-04-07 09:53:46,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=6870, suffix heap size=151312,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=150 2023-04-07 09:53:46,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680861226139, max timestamp=1680861226162Num uniques 1095; 2023-04-07 09:53:46,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-07 09:53:46,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680861226139, max timestamp=1680861226162Num uniques 1009; 2023-04-07 09:53:46,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680861226139, max timestamp=1680861226162Num uniques 1095; 2023-04-07 09:53:46,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-07 09:53:46,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-07 09:53:46,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861286240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861286241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861286253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861286260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861286260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,288 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/7b5513d5d7194a19b887a8ee9c03f728 2023-04-07 09:53:46,339 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-07 09:53:46,343 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-07 09:53:46,345 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-07 09:53:46,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861286358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861286359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861286362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861286364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861286364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,448 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/a02cf64048da4621b5dc17d7de4aa1cb 2023-04-07 09:53:46,510 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/bf96b3e5047442d48b76322df0193a4c 2023-04-07 09:53:46,535 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/7b5513d5d7194a19b887a8ee9c03f728 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7b5513d5d7194a19b887a8ee9c03f728 2023-04-07 09:53:46,548 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7b5513d5d7194a19b887a8ee9c03f728, entries=150, sequenceid=9, filesize=11.6 K 2023-04-07 09:53:46,557 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-07 09:53:46,568 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/a02cf64048da4621b5dc17d7de4aa1cb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a02cf64048da4621b5dc17d7de4aa1cb 2023-04-07 09:53:46,591 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a02cf64048da4621b5dc17d7de4aa1cb, entries=150, sequenceid=9, filesize=11.6 K 2023-04-07 09:53:46,594 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/bf96b3e5047442d48b76322df0193a4c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bf96b3e5047442d48b76322df0193a4c 2023-04-07 09:53:46,610 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bf96b3e5047442d48b76322df0193a4c, entries=150, sequenceid=9, filesize=11.6 K 2023-04-07 09:53:46,613 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 0b39eaaa997d54d62464302ce4e71f9b in 937ms, sequenceid=9, compaction requested=false 2023-04-07 09:53:46,616 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:46,617 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:46,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:46,622 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:46,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,632 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:46,641 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce7ece4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:46,641 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,643 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:46,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=154.31 KB heapSize=437.39 KB 2023-04-07 09:53:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:46,651 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:46,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:46,652 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:46,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:46,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:46,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:46,652 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:46,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:46,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:53:46,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:46,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:53:46,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:46,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:53:46,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:46,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,659 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:46,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861286637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,690 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:46,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:46,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:46,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:46,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:46,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:46,696 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:46,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:46,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:46,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:46,700 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:46,700 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:46,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861286654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861286664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861286676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861286668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861286773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:46,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:46,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861286811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861286811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:46,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:46,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:46,822 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:46,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:46,824 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:46,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:46,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:46,827 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:46,827 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:46,827 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:46,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:46,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:46,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:46,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:46,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:46,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:46,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:46,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:46,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:46,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:46,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e56514[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:46,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:46,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:46,846 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,846 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:46,849 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:46,850 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:46,850 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:46,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:46,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861286821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861286825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:46,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:46,871 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:46,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:46,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:46,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:46,871 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:46,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:46,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:46,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,885 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:46,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:46,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:46,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:46,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:46,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:46,901 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,901 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:46,901 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:46,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:46,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:46,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:46,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:46,903 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:46,903 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:46,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:46,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:46,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:46,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861286995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,003 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:47,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:47,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:47,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861287016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861287018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:47,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:47,024 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:47,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:47,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:47,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:47,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:47,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:47,036 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:47,036 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:47,037 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:47,037 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,037 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:47,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:47,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:47,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,041 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15795fb4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:47,045 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:47,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:47,047 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,047 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:47,057 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:47,057 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,057 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:47,058 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:47,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:47,062 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:47,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:47,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:47,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:47,063 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:47,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,070 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861287072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861287073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,091 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 09:53:47,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:47,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:47,094 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:47,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,094 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:47,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/66c09c8b213e45458bf903883a024d2b 2023-04-07 09:53:47,194 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:47,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:47,198 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-07 09:53:47,199 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-07 09:53:47,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:47,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:47,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:47,217 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:47,217 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:47,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:47,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:47,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:47,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:47,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:47,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:47,242 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,242 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:47,242 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:47,242 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:47,246 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:47,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59b1d50e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:47,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:47,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,253 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:47,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:47,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:47,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:47,262 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:47,262 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:47,262 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:47,262 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:47,262 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:47,265 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,265 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,265 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,266 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:47,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:47,284 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:47,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:47,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:47,285 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,285 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861287300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861287326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861287325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861287381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:47,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:47,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:47,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861287385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:47,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:47,399 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:47,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,405 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:47,405 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:47,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,409 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:47,411 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,411 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,411 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:47,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:47,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:47,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:47,413 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,413 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:47,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:47,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:47,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:47,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,418 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a229210[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:47,420 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:47,420 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,422 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:47,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:47,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:47,424 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,424 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:47,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:47,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:47,435 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:47,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:47,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:47,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:47,435 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:47,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,439 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,440 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,442 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:47,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,448 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:47,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:47,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,549 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:47,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:47,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:47,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:47,573 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:47,573 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,575 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:47,575 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:47,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,578 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,578 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:47,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:47,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:47,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,588 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:47,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:47,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:47,598 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,598 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:47,598 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:47,598 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:47,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@381d9385[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:47,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:47,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,613 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:47,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,615 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:47,618 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:47,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:47,625 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:47,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:47,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:47,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:47,625 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:47,628 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,628 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,628 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:47,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:47,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:47,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/7cea7b6ac6354e95b3cc0e722096d75e 2023-04-07 09:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,640 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:47,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:47,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:47,640 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:47,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:47,641 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:47,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/1c5838df8fd344e591b4ab74fe7ab390 2023-04-07 09:53:47,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/66c09c8b213e45458bf903883a024d2b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/66c09c8b213e45458bf903883a024d2b 2023-04-07 09:53:47,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/66c09c8b213e45458bf903883a024d2b, entries=250, sequenceid=117, filesize=16.2 K 2023-04-07 09:53:47,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/7cea7b6ac6354e95b3cc0e722096d75e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7cea7b6ac6354e95b3cc0e722096d75e 2023-04-07 09:53:47,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7cea7b6ac6354e95b3cc0e722096d75e, entries=150, sequenceid=117, filesize=11.6 K 2023-04-07 09:53:47,751 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:47,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/1c5838df8fd344e591b4ab74fe7ab390 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1c5838df8fd344e591b4ab74fe7ab390 2023-04-07 09:53:47,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:47,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:47,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:47,764 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:47,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,765 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:47,765 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:47,767 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:47,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1c5838df8fd344e591b4ab74fe7ab390, entries=150, sequenceid=117, filesize=11.6 K 2023-04-07 09:53:47,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:47,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:47,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:47,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~457.27 KB/468240, currentSize=20.13 KB/20610 for 0b39eaaa997d54d62464302ce4e71f9b in 1122ms, sequenceid=117, compaction requested=false 2023-04-07 09:53:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:47,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:47,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:47,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:47,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:47,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:47,780 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:47,780 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:47,781 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-07 09:53:47,781 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:47,781 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:47,781 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:47,783 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:47,783 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:47,783 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:47,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:47,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:47,846 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/779c957179474c60aaaec851f39f2e88 2023-04-07 09:53:47,888 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/87044303560b45518e7cb5c7a98cfd10 2023-04-07 09:53:47,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861227847, max timestamp=1680861227899Num uniques 859; 2023-04-07 09:53:47,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:53:47,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861227847, max timestamp=1680861227900Num uniques 859; 2023-04-07 09:53:47,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:53:47,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861227847, max timestamp=1680861227899Num uniques 859; 2023-04-07 09:53:47,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:53:47,945 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/1d56fb8b0feb487fb24f18f91d21a9ed 2023-04-07 09:53:47,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861227942, max timestamp=1680861227950Num uniques 1068; 2023-04-07 09:53:47,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861227942, max timestamp=1680861227950Num uniques 1068; 2023-04-07 09:53:47,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:53:47,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:53:47,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861227942, max timestamp=1680861227950Num uniques 1068; 2023-04-07 09:53:47,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:53:47,959 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/779c957179474c60aaaec851f39f2e88 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/779c957179474c60aaaec851f39f2e88 2023-04-07 09:53:47,971 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/779c957179474c60aaaec851f39f2e88, entries=100, sequenceid=123, filesize=9.3 K 2023-04-07 09:53:47,974 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/87044303560b45518e7cb5c7a98cfd10 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/87044303560b45518e7cb5c7a98cfd10 2023-04-07 09:53:47,988 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/87044303560b45518e7cb5c7a98cfd10, entries=100, sequenceid=123, filesize=9.3 K 2023-04-07 09:53:47,992 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/1d56fb8b0feb487fb24f18f91d21a9ed as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1d56fb8b0feb487fb24f18f91d21a9ed 2023-04-07 09:53:47,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=527, cellSize=24132, totalHeapSize=55048, min timestamp=1680861227955, max timestamp=1680861227994Num uniques 1027; 2023-04-07 09:53:48,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=24132, suffix heap size=134776,new segment heap size=55048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=527 2023-04-07 09:53:47,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=527, cellSize=24132, totalHeapSize=55048, min timestamp=1680861227955, max timestamp=1680861227994Num uniques 1027; 2023-04-07 09:53:48,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=527, cellSize=24132, totalHeapSize=55048, min timestamp=1680861227955, max timestamp=1680861227994Num uniques 1027; 2023-04-07 09:53:48,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=24132, suffix heap size=134776,new segment heap size=55048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=527 2023-04-07 09:53:48,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=24132, suffix heap size=134776,new segment heap size=55048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=527 2023-04-07 09:53:48,013 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1d56fb8b0feb487fb24f18f91d21a9ed, entries=100, sequenceid=123, filesize=9.3 K 2023-04-07 09:53:48,015 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=100.63 KB/103050 for 0b39eaaa997d54d62464302ce4e71f9b in 233ms, sequenceid=123, compaction requested=true 2023-04-07 09:53:48,015 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:48,015 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:48,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:48,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=107.34 KB heapSize=270.80 KB 2023-04-07 09:53:48,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:48,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:48,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:48,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:48,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:48,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:48,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67c6ffcd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:48,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:48,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,031 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,031 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:48,031 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:48,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:48,035 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:48,039 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:48,039 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:48,039 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:48,039 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:48,039 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:48,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,051 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,060 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/8e3c8337d8714fd4b872c5cdf24f2e7f 2023-04-07 09:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,077 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,077 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:48,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:48,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,079 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:48,079 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:48,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861288073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861288075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861288077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861288107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861288109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,149 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/117a7728c3334c87a2247a91fc3a0fb3 2023-04-07 09:53:48,179 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:48,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:48,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:48,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/2cc01b012bdb424188b756480dd329e8 2023-04-07 09:53:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:48,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:48,212 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:48,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,213 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:48,213 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:48,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,215 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:48,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:48,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861288214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861288215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/8e3c8337d8714fd4b872c5cdf24f2e7f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/8e3c8337d8714fd4b872c5cdf24f2e7f 2023-04-07 09:53:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861288218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:48,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:48,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:48,235 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861288229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/8e3c8337d8714fd4b872c5cdf24f2e7f, entries=250, sequenceid=184, filesize=16.4 K 2023-04-07 09:53:48,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/117a7728c3334c87a2247a91fc3a0fb3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/117a7728c3334c87a2247a91fc3a0fb3 2023-04-07 09:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:48,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:48,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:48,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:48,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30085f9e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:48,258 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:48,258 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:48,263 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,263 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:48,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861288252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,266 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:48,266 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:48,267 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:48,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:48,274 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:48,274 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:48,274 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:48,274 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:48,274 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:48,276 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,276 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,276 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/117a7728c3334c87a2247a91fc3a0fb3, entries=150, sequenceid=184, filesize=11.8 K 2023-04-07 09:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,279 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/2cc01b012bdb424188b756480dd329e8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/2cc01b012bdb424188b756480dd329e8 2023-04-07 09:53:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,288 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:48,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:48,289 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,293 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,293 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,293 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,293 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,294 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:48,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:48,294 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:48,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/2cc01b012bdb424188b756480dd329e8, entries=150, sequenceid=184, filesize=11.8 K 2023-04-07 09:53:48,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~290.67 KB/297648, currentSize=87.22 KB/89310 for 0b39eaaa997d54d62464302ce4e71f9b in 293ms, sequenceid=184, compaction requested=true 2023-04-07 09:53:48,313 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:48,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:48,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:48,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:48,321 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:48,321 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:48,333 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54810 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:48,333 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:48,335 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:48,335 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,335 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:48,336 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,336 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a02cf64048da4621b5dc17d7de4aa1cb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7cea7b6ac6354e95b3cc0e722096d75e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/87044303560b45518e7cb5c7a98cfd10, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/117a7728c3334c87a2247a91fc3a0fb3] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=44.3 K 2023-04-07 09:53:48,336 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7b5513d5d7194a19b887a8ee9c03f728, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/66c09c8b213e45458bf903883a024d2b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/779c957179474c60aaaec851f39f2e88, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/8e3c8337d8714fd4b872c5cdf24f2e7f] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=53.5 K 2023-04-07 09:53:48,337 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a02cf64048da4621b5dc17d7de4aa1cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680861225642 2023-04-07 09:53:48,339 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7cea7b6ac6354e95b3cc0e722096d75e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680861226232 2023-04-07 09:53:48,341 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 87044303560b45518e7cb5c7a98cfd10, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680861226655 2023-04-07 09:53:48,341 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 7b5513d5d7194a19b887a8ee9c03f728, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680861225642 2023-04-07 09:53:48,345 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 117a7728c3334c87a2247a91fc3a0fb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680861228010 2023-04-07 09:53:48,346 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 66c09c8b213e45458bf903883a024d2b, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680861226232 2023-04-07 09:53:48,347 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 779c957179474c60aaaec851f39f2e88, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680861226655 2023-04-07 09:53:48,352 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8e3c8337d8714fd4b872c5cdf24f2e7f, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680861228003 2023-04-07 09:53:48,372 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#12 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:48,389 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#13 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:48,394 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:48,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:48,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-07 09:53:48,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:48,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:48,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:48,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:48,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:48,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:48,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:48,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:48,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:48,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:48,444 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:48,444 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:48,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:48,447 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,447 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,449 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,450 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,450 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:48,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:48,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:48,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:48,459 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,459 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:48,460 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:48,460 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,484 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/14acc2cff502435bbeba80019405b5d0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/14acc2cff502435bbeba80019405b5d0 2023-04-07 09:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:48,484 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/51b095144ab4406ebecaab01c88c7cb3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/51b095144ab4406ebecaab01c88c7cb3 2023-04-07 09:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee4f99d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:48,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:48,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:48,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:48,497 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:48,503 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:48,503 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:48,503 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:48,503 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:48,503 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:48,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:48,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:48,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,513 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,517 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:48,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,517 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:48,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/fcf8cb982f2d4469b48619324a12c4ca 2023-04-07 09:53:48,529 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into 51b095144ab4406ebecaab01c88c7cb3(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:48,529 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:48,529 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=12, startTime=1680861228320; duration=0sec 2023-04-07 09:53:48,530 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 14acc2cff502435bbeba80019405b5d0(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:48,530 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:48,530 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=12, startTime=1680861228320; duration=0sec 2023-04-07 09:53:48,530 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:48,530 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:48,530 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:48,541 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:48,541 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:48,541 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,541 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bf96b3e5047442d48b76322df0193a4c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1c5838df8fd344e591b4ab74fe7ab390, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1d56fb8b0feb487fb24f18f91d21a9ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/2cc01b012bdb424188b756480dd329e8] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=44.3 K 2023-04-07 09:53:48,543 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bf96b3e5047442d48b76322df0193a4c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680861225642 2023-04-07 09:53:48,543 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1c5838df8fd344e591b4ab74fe7ab390, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680861226232 2023-04-07 09:53:48,549 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1d56fb8b0feb487fb24f18f91d21a9ed, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680861226655 2023-04-07 09:53:48,549 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 2cc01b012bdb424188b756480dd329e8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680861228010 2023-04-07 09:53:48,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861288542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861288535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861288543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861288557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861288557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,591 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#16 average throughput is 0.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:48,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/d1ed7bfa2a444fe5b65bfaf72d23750d 2023-04-07 09:53:48,618 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:48,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:48,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861288662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861288665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861288665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861288665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861288668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:48,672 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:48,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:48,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:48,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:48,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:48,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:48,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:48,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:48,692 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,693 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:48,693 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:48,693 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:48,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:48,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:48,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:48,696 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,696 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a31ce7f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:48,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:48,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:48,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:48,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:48,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,702 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:48,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:48,706 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:48,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:48,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:48,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:48,706 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:48,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,734 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,734 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:48,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:48,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:48,734 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,735 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:48,735 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,835 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:48,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:48,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:48,844 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:48,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:48,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:48,846 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,846 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,847 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,848 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,848 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:48,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:48,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:48,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:48,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:48,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:48,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:48,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:48,864 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:48,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad96663[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:48,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:48,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,870 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:48,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861288871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861288871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861288872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861288872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:48,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861288872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:48,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:48,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:48,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:48,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:48,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:48,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:48,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:48,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,885 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:48,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:48,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:48,888 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:48,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:48,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:48,889 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:48,890 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:48,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:48,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:48,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:48,890 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:48,890 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:48,990 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:48,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:48,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:48,999 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:48,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:49,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:49,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:49,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:49,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:49,004 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:49,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:49,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:49,006 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:49,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:49,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,008 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:49,008 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:49,008 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:49,008 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:49,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:49,013 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,014 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,014 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:49,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185f2009[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:49,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:49,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,020 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:49,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:49,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:49,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:49,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:49,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:49,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:49,027 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:49,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:49,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:49,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:49,027 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:49,031 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,031 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,031 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:49,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,033 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,037 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:49,037 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:49,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:49,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:49,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:49,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:49,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:49,039 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:49,039 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:49,074 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/b7b6b22a7e864007b687972d5b38baca as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b7b6b22a7e864007b687972d5b38baca 2023-04-07 09:53:49,085 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into b7b6b22a7e864007b687972d5b38baca(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:49,086 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:49,087 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=12, startTime=1680861228320; duration=0sec 2023-04-07 09:53:49,087 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:49,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:49,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:49,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:49,145 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:49,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:49,146 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:49,146 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:49,148 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,148 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,149 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:49,149 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:49,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:49,153 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/b2690cb370fd468b83c603646628b481 2023-04-07 09:53:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:49,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:49,157 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:49,157 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:49,157 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:49,160 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:49,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513e0a8e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:49,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:49,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,169 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:49,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:49,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/fcf8cb982f2d4469b48619324a12c4ca as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/fcf8cb982f2d4469b48619324a12c4ca 2023-04-07 09:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:49,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:49,182 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:49,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:49,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:49,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:49,182 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:49,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861289181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861289182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861289182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861289184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,189 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861289185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/fcf8cb982f2d4469b48619324a12c4ca, entries=200, sequenceid=201, filesize=14.1 K 2023-04-07 09:53:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,201 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/d1ed7bfa2a444fe5b65bfaf72d23750d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d1ed7bfa2a444fe5b65bfaf72d23750d 2023-04-07 09:53:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:49,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:49,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,206 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:49,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:49,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:49,212 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:49,213 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:49,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:49,213 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:49,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d1ed7bfa2a444fe5b65bfaf72d23750d, entries=150, sequenceid=201, filesize=11.8 K 2023-04-07 09:53:49,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/b2690cb370fd468b83c603646628b481 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b2690cb370fd468b83c603646628b481 2023-04-07 09:53:49,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b2690cb370fd468b83c603646628b481, entries=150, sequenceid=201, filesize=11.8 K 2023-04-07 09:53:49,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 0b39eaaa997d54d62464302ce4e71f9b in 801ms, sequenceid=201, compaction requested=false 2023-04-07 09:53:49,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:49,313 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:49,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:49,319 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:49,319 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:49,321 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:49,321 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:49,321 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,321 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:49,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:49,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:49,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:49,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:49,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:49,327 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:49,327 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:49,327 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-07 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:49,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:49,374 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/565415bf02f949dc814d506b518709db 2023-04-07 09:53:49,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:49,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:49,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861289731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861289732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861289736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861289737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861289738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,841 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/b7120c8b81694b1199ebbac71a1c1cd7 2023-04-07 09:53:49,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861289840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861289841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861289846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861289846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:49,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861289847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,910 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/a4c24100c38d4e29b5d153f0073c7afa 2023-04-07 09:53:49,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/565415bf02f949dc814d506b518709db as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/565415bf02f949dc814d506b518709db 2023-04-07 09:53:49,943 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/565415bf02f949dc814d506b518709db, entries=150, sequenceid=220, filesize=11.8 K 2023-04-07 09:53:49,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/b7120c8b81694b1199ebbac71a1c1cd7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/b7120c8b81694b1199ebbac71a1c1cd7 2023-04-07 09:53:49,958 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/b7120c8b81694b1199ebbac71a1c1cd7, entries=150, sequenceid=220, filesize=11.8 K 2023-04-07 09:53:49,960 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/a4c24100c38d4e29b5d153f0073c7afa as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/a4c24100c38d4e29b5d153f0073c7afa 2023-04-07 09:53:49,975 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/a4c24100c38d4e29b5d153f0073c7afa, entries=150, sequenceid=220, filesize=11.8 K 2023-04-07 09:53:49,976 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0b39eaaa997d54d62464302ce4e71f9b in 649ms, sequenceid=220, compaction requested=true 2023-04-07 09:53:49,977 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:49,977 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:49,980 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:49,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c864e0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:49,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:49,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:49,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:49,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:49,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:49,992 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:49,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:49,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:49,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:49,992 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:49,995 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,995 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:49,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,995 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:49,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:49,996 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,998 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:50,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:50,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:50,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:50,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:50,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:50,050 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 09:53:50,050 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:50,050 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:50,050 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:50,050 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:50,050 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:50,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:50,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861290070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861290072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861290074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861290076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861290084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,104 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:50,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:50,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:50,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:50,144 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:50,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:50,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:50,146 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,146 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,150 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,150 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,150 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,150 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:50,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:50,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:50,157 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4403fc44[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:50,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:50,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:50,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:50,167 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:50,176 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:50,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:50,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:50,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:50,177 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:50,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861290185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,190 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,190 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,190 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,191 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,190 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:50,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861290186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861290189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861290190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861290186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:50,200 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:50,200 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:50,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:50,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:50,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:50,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:50,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:50,306 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:50,307 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:50,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:50,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:50,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:50,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:50,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:50,316 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:50,316 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:50,316 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,316 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:50,316 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:50,317 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:50,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:50,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:50,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:50,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@253dc3a5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:50,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:50,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,323 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:50,323 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:50,323 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:50,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:50,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:50,335 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:50,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:50,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:50,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:50,335 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:50,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,346 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,346 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,347 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,347 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:50,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:50,347 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,347 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,349 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,349 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,349 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,349 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,348 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:50,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:50,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861290390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861290395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861290395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861290396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861290401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:50,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:50,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:50,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:50,455 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:50,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:50,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:50,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:50,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:50,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:50,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:50,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:50,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476d7f6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:50,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:50,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:50,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:50,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:50,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:50,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:50,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:50,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:50,490 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:50,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:50,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:50,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:50,490 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:50,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,493 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,495 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:50,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:50,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,505 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:50,505 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:50,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:50,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/7991f9fb5edf421a97dde03de5b5afc0 2023-04-07 09:53:50,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/c25a79ce98c544c699d6a91dbe41ef5e 2023-04-07 09:53:50,605 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:50,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:50,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:50,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:50,628 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:50,628 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:50,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:50,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,633 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/f88c33938b724287898eaf614709adc8 2023-04-07 09:53:50,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,635 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,635 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:50,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:50,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/7991f9fb5edf421a97dde03de5b5afc0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7991f9fb5edf421a97dde03de5b5afc0 2023-04-07 09:53:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22bf5cfc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:50,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:50,650 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,651 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:50,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:50,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:50,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:50,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7991f9fb5edf421a97dde03de5b5afc0, entries=150, sequenceid=238, filesize=11.8 K 2023-04-07 09:53:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:50,658 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:50,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:50,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:50,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:50,658 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:50,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/c25a79ce98c544c699d6a91dbe41ef5e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c25a79ce98c544c699d6a91dbe41ef5e 2023-04-07 09:53:50,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:50,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:50,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:50,668 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:50,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:50,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:50,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:50,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,673 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:50,674 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,674 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:50,674 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:50,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:50,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:50,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c25a79ce98c544c699d6a91dbe41ef5e, entries=150, sequenceid=238, filesize=11.8 K 2023-04-07 09:53:50,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/f88c33938b724287898eaf614709adc8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/f88c33938b724287898eaf614709adc8 2023-04-07 09:53:50,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861290695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861290700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861290700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861290701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:50,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861290705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/f88c33938b724287898eaf614709adc8, entries=150, sequenceid=238, filesize=11.8 K 2023-04-07 09:53:50,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0b39eaaa997d54d62464302ce4e71f9b in 665ms, sequenceid=238, compaction requested=true 2023-04-07 09:53:50,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:50,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:50,715 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:50,716 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:50,716 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:50,716 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:50,719 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:50,719 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:50,719 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,720 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/14acc2cff502435bbeba80019405b5d0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d1ed7bfa2a444fe5b65bfaf72d23750d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/b7120c8b81694b1199ebbac71a1c1cd7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c25a79ce98c544c699d6a91dbe41ef5e] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=47.2 K 2023-04-07 09:53:50,720 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50672 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:50,720 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:50,720 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,720 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/51b095144ab4406ebecaab01c88c7cb3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/fcf8cb982f2d4469b48619324a12c4ca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/565415bf02f949dc814d506b518709db, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7991f9fb5edf421a97dde03de5b5afc0] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=49.5 K 2023-04-07 09:53:50,721 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 14acc2cff502435bbeba80019405b5d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680861228010 2023-04-07 09:53:50,722 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d1ed7bfa2a444fe5b65bfaf72d23750d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680861228070 2023-04-07 09:53:50,722 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 51b095144ab4406ebecaab01c88c7cb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680861228010 2023-04-07 09:53:50,723 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b7120c8b81694b1199ebbac71a1c1cd7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680861228452 2023-04-07 09:53:50,724 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting fcf8cb982f2d4469b48619324a12c4ca, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680861228070 2023-04-07 09:53:50,724 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c25a79ce98c544c699d6a91dbe41ef5e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680861229734 2023-04-07 09:53:50,725 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 565415bf02f949dc814d506b518709db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680861228452 2023-04-07 09:53:50,725 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 7991f9fb5edf421a97dde03de5b5afc0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680861229734 2023-04-07 09:53:50,753 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:50,768 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#25 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:50,775 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:50,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:50,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:50,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:50,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:50,789 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:50,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,793 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:50,793 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:50,794 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,795 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:50,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:50,798 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:50,798 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:50,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:50,804 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/70ae3778a5e74344a339567e37491966 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/70ae3778a5e74344a339567e37491966 2023-04-07 09:53:50,817 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 70ae3778a5e74344a339567e37491966(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:50,818 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:50,818 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=12, startTime=1680861230715; duration=0sec 2023-04-07 09:53:50,818 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:50,818 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:50,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:50,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:50,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:50,826 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:50,828 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:50,828 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:50,828 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,828 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b7b6b22a7e864007b687972d5b38baca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b2690cb370fd468b83c603646628b481, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/a4c24100c38d4e29b5d153f0073c7afa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/f88c33938b724287898eaf614709adc8] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=47.2 K 2023-04-07 09:53:50,829 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b7b6b22a7e864007b687972d5b38baca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680861228010 2023-04-07 09:53:50,829 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b2690cb370fd468b83c603646628b481, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680861228070 2023-04-07 09:53:50,830 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a4c24100c38d4e29b5d153f0073c7afa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680861228452 2023-04-07 09:53:50,830 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f88c33938b724287898eaf614709adc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680861229734 2023-04-07 09:53:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:50,832 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:50,837 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:50,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:50,837 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 09:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:50,930 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#27 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:51,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:51,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:51,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861291223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861291225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861291226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861291228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861291228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,250 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/c5d30d1c02184cfea0a69a486ba83bb2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/c5d30d1c02184cfea0a69a486ba83bb2 2023-04-07 09:53:51,262 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into c5d30d1c02184cfea0a69a486ba83bb2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:51,262 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:51,262 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=12, startTime=1680861230715; duration=0sec 2023-04-07 09:53:51,262 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:51,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861291331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861291331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861291336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861291337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861291338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,367 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/08f611ac931b4ced825d3d0d8f7ec144 2023-04-07 09:53:51,401 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/bfb051901c9d4ac2a244175d94e45169 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bfb051901c9d4ac2a244175d94e45169 2023-04-07 09:53:51,413 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into bfb051901c9d4ac2a244175d94e45169(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:51,413 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:51,413 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=12, startTime=1680861230716; duration=0sec 2023-04-07 09:53:51,413 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:51,445 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/fb8e65500b7148a5ba62bf0555aa4313 2023-04-07 09:53:51,480 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/12604177e5ac4487af4bdf23b0116492 2023-04-07 09:53:51,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/08f611ac931b4ced825d3d0d8f7ec144 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/08f611ac931b4ced825d3d0d8f7ec144 2023-04-07 09:53:51,498 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/08f611ac931b4ced825d3d0d8f7ec144, entries=150, sequenceid=253, filesize=11.8 K 2023-04-07 09:53:51,501 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/fb8e65500b7148a5ba62bf0555aa4313 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/fb8e65500b7148a5ba62bf0555aa4313 2023-04-07 09:53:51,511 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/fb8e65500b7148a5ba62bf0555aa4313, entries=150, sequenceid=253, filesize=11.8 K 2023-04-07 09:53:51,518 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/12604177e5ac4487af4bdf23b0116492 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/12604177e5ac4487af4bdf23b0116492 2023-04-07 09:53:51,526 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/12604177e5ac4487af4bdf23b0116492, entries=150, sequenceid=253, filesize=11.8 K 2023-04-07 09:53:51,529 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0b39eaaa997d54d62464302ce4e71f9b in 692ms, sequenceid=253, compaction requested=false 2023-04-07 09:53:51,529 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:51,529 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:51,529 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:51,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:51,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:51,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:51,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,534 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79056533[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:51,537 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:51,537 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,538 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:51,539 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,539 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:51,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:51,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:53:51,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:51,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:51,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:51,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:51,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:51,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:51,545 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:51,545 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:51,553 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:51,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:51,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:51,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:51,553 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:51,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:51,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:51,556 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,564 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:51,564 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:51,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:51,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:51,565 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:51,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:51,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:51,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,566 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:51,566 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:51,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861291558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861291561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861291564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,576 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/5284a66e109d46f291319eb10099b58d 2023-04-07 09:53:51,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861291568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861291566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/9017511772364f8198865310b3f2e066 2023-04-07 09:53:51,666 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:51,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:51,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:51,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:51,674 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:51,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:51,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:51,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:51,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861291671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861291672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:51,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:51,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:51,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861291677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:51,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861291678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:51,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:51,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:51,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:51,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:51,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:51,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:51,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861291679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,687 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32eea15d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:51,689 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:51,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:51,694 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:51,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:51,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:51,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,696 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:51,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:51,706 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:51,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:51,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:51,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:51,706 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:51,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:51,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:51,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:51,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:51,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:51,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:51,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:51,723 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:51,723 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:51,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:51,823 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:51,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:51,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:51,830 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:51,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:51,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:51,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:51,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:51,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:51,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:51,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:51,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:51,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:51,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:51,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:51,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@951991[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:51,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:51,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,846 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:51,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:51,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,850 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:51,850 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:51,850 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:51,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:51,858 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:51,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:51,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:51,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:51,858 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:51,860 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,860 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,860 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,864 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:51,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:51,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:51,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:51,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:51,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:51,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:51,873 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:51,872 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:51,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:51,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861291877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861291879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861291881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861291881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:51,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861291888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,973 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:51,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:51,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:51,978 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:51,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:51,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:51,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:51,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,980 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:51,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:51,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:51,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:51,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:51,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:51,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:51,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:51,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:51,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:51,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:51,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:51,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:51,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:51,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5815fdc3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:51,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:51,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:51,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:51,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:51,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:51,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:51,992 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:51,992 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,992 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,992 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:51,994 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:51,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:51,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:51,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:51,995 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:51,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:51,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:51,998 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,005 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:52,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:52,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,006 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,006 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,006 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,006 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,006 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:52,006 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:52,006 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/0a5f83d7d46f43e09ae2299fd7927496 2023-04-07 09:53:52,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/5284a66e109d46f291319eb10099b58d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5284a66e109d46f291319eb10099b58d 2023-04-07 09:53:52,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5284a66e109d46f291319eb10099b58d, entries=150, sequenceid=275, filesize=11.9 K 2023-04-07 09:53:52,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/9017511772364f8198865310b3f2e066 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/9017511772364f8198865310b3f2e066 2023-04-07 09:53:52,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/9017511772364f8198865310b3f2e066, entries=150, sequenceid=275, filesize=11.9 K 2023-04-07 09:53:52,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/0a5f83d7d46f43e09ae2299fd7927496 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/0a5f83d7d46f43e09ae2299fd7927496 2023-04-07 09:53:52,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/0a5f83d7d46f43e09ae2299fd7927496, entries=150, sequenceid=275, filesize=11.9 K 2023-04-07 09:53:52,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0b39eaaa997d54d62464302ce4e71f9b in 551ms, sequenceid=275, compaction requested=true 2023-04-07 09:53:52,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:52,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:52,094 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:52,094 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:52,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:52,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:52,096 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36532 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:52,096 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:52,096 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,096 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/c5d30d1c02184cfea0a69a486ba83bb2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/08f611ac931b4ced825d3d0d8f7ec144, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5284a66e109d46f291319eb10099b58d] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=35.7 K 2023-04-07 09:53:52,096 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36532 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:52,096 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:52,097 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,097 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/70ae3778a5e74344a339567e37491966, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/fb8e65500b7148a5ba62bf0555aa4313, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/9017511772364f8198865310b3f2e066] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=35.7 K 2023-04-07 09:53:52,098 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 70ae3778a5e74344a339567e37491966, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680861229734 2023-04-07 09:53:52,099 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting c5d30d1c02184cfea0a69a486ba83bb2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680861229734 2023-04-07 09:53:52,101 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fb8e65500b7148a5ba62bf0555aa4313, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680861230072 2023-04-07 09:53:52,101 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 08f611ac931b4ced825d3d0d8f7ec144, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680861230072 2023-04-07 09:53:52,101 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9017511772364f8198865310b3f2e066, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861231223 2023-04-07 09:53:52,102 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5284a66e109d46f291319eb10099b58d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861231223 2023-04-07 09:53:52,106 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:52,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:52,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:52,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:52,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:52,120 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:52,120 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,120 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:52,120 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:52,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,123 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:52,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:52,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:52,130 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:52,130 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:52,130 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#33 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:52,130 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,131 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:52,132 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:53:52,132 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:52,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:52,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:52,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:52,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:52,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:52,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:52,164 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/d702393a707a40b895bb2bc47403d9ed as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/d702393a707a40b895bb2bc47403d9ed 2023-04-07 09:53:52,177 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/0896455ab8f842e99efa8ac5af7d3dc2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0896455ab8f842e99efa8ac5af7d3dc2 2023-04-07 09:53:52,177 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into d702393a707a40b895bb2bc47403d9ed(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:52,177 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:52,178 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=13, startTime=1680861232093; duration=0sec 2023-04-07 09:53:52,184 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/ea9df95c335a4b52b2f95bf970d85e0d 2023-04-07 09:53:52,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:52,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:52,186 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:52,189 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:52,192 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36532 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:52,192 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:52,193 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,193 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bfb051901c9d4ac2a244175d94e45169, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/12604177e5ac4487af4bdf23b0116492, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/0a5f83d7d46f43e09ae2299fd7927496] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=35.7 K 2023-04-07 09:53:52,194 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 0896455ab8f842e99efa8ac5af7d3dc2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:52,194 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:52,194 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=13, startTime=1680861232094; duration=0sec 2023-04-07 09:53:52,194 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:52,194 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bfb051901c9d4ac2a244175d94e45169, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680861229734 2023-04-07 09:53:52,197 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 12604177e5ac4487af4bdf23b0116492, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680861230072 2023-04-07 09:53:52,198 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 0a5f83d7d46f43e09ae2299fd7927496, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861231223 2023-04-07 09:53:52,217 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:52,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861292217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861292217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861292218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861292226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861292224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,232 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/1216b87b20ee4c21b34f2f92c94255e4 2023-04-07 09:53:52,251 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/5ad6ffcab34649c9bf9254d4a93c2a60 2023-04-07 09:53:52,259 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/ea9df95c335a4b52b2f95bf970d85e0d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ea9df95c335a4b52b2f95bf970d85e0d 2023-04-07 09:53:52,268 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ea9df95c335a4b52b2f95bf970d85e0d, entries=150, sequenceid=287, filesize=11.9 K 2023-04-07 09:53:52,269 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/1216b87b20ee4c21b34f2f92c94255e4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1216b87b20ee4c21b34f2f92c94255e4 2023-04-07 09:53:52,280 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1216b87b20ee4c21b34f2f92c94255e4, entries=150, sequenceid=287, filesize=11.9 K 2023-04-07 09:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/5ad6ffcab34649c9bf9254d4a93c2a60 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ad6ffcab34649c9bf9254d4a93c2a60 2023-04-07 09:53:52,287 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ad6ffcab34649c9bf9254d4a93c2a60, entries=150, sequenceid=287, filesize=11.9 K 2023-04-07 09:53:52,288 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0b39eaaa997d54d62464302ce4e71f9b in 156ms, sequenceid=287, compaction requested=false 2023-04-07 09:53:52,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:52,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,288 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:52,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:52,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:52,292 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:52,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a6b370f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:52,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:52,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,297 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:52,298 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:52,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:52,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:52,299 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:52,303 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:52,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:52,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:52,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:52,303 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:52,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,307 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,311 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,311 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,312 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:52,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:52,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:52,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,312 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:52,312 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:52,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-07 09:53:52,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861232212, max timestamp=1680861232329Num uniques 859; 2023-04-07 09:53:52,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:53:52,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861232212, max timestamp=1680861232329Num uniques 859; 2023-04-07 09:53:52,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:53:52,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861232212, max timestamp=1680861232329Num uniques 859; 2023-04-07 09:53:52,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:53:52,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:52,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:52,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:52,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:52,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:52,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:52,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861232345, max timestamp=1680861232356Num uniques 859; 2023-04-07 09:53:52,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:53:52,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861232345, max timestamp=1680861232356Num uniques 859; 2023-04-07 09:53:52,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861232345, max timestamp=1680861232356Num uniques 859; 2023-04-07 09:53:52,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:53:52,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:53:52,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680861232370, max timestamp=1680861232386Num uniques 1118; 2023-04-07 09:53:52,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-07 09:53:52,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680861232370, max timestamp=1680861232386Num uniques 1118; 2023-04-07 09:53:52,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-07 09:53:52,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680861232370, max timestamp=1680861232386Num uniques 1118; 2023-04-07 09:53:52,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-07 09:53:52,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:52,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:52,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:52,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680861232401, max timestamp=1680861232417Num uniques 1127; 2023-04-07 09:53:52,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680861232401, max timestamp=1680861232417Num uniques 1127; 2023-04-07 09:53:52,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-07 09:53:52,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-07 09:53:52,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861232407, max timestamp=1680861232417Num uniques 1127; 2023-04-07 09:53:52,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=6870, suffix heap size=145176,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=150 2023-04-07 09:53:52,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:52,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:52,432 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:52,432 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,432 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:52,432 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:52,433 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,433 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,434 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,434 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:52,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:52,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:52,437 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:52,437 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,437 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:52,440 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,444 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:52,444 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:52,444 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:52,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:52,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d488b56[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:52,451 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:52,451 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,453 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:52,454 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:52,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861292449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680861232441, max timestamp=1680861232460Num uniques 1236; 2023-04-07 09:53:52,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=15386, suffix heap size=156512,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=336 2023-04-07 09:53:52,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680861232441, max timestamp=1680861232460Num uniques 1236; 2023-04-07 09:53:52,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=15386, suffix heap size=156512,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=336 2023-04-07 09:53:52,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680861232441, max timestamp=1680861232460Num uniques 1009; 2023-04-07 09:53:52,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15386, suffix heap size=132904,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=336 2023-04-07 09:53:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:52,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:52,472 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:52,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:52,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:52,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:52,472 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:52,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:52,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:52,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:52,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861292538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57324 deadline: 1680861292540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861292548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861292548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861292565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,592 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:52,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:52,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:52,617 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:52,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:52,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:52,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,621 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:52,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:52,648 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:52,648 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:52,648 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:52,649 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:52,652 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861292650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:52,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57350 deadline: 1680861292649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:52,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65b94aff[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:52,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:52,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,657 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,658 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:52,658 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:52,658 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:52,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:52,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57336 deadline: 1680861292653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,659 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,659 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:52,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861292654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,669 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/bb68bc95db2f4dde88ebfee99f1822b9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bb68bc95db2f4dde88ebfee99f1822b9 2023-04-07 09:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:52,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:52,676 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:52,676 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:52,676 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:52,676 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:52,676 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:52,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,682 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into bb68bc95db2f4dde88ebfee99f1822b9(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,682 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:52,682 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=13, startTime=1680861232094; duration=0sec 2023-04-07 09:53:52,683 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,683 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:52,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:52,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,692 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,693 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:52,693 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:52,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/eef9f380433d4ea0ba3807fdced6c514 2023-04-07 09:53:52,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861292772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,793 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:52,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:52,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:52,828 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:52,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:52,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:52,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:52,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:52,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:52,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:52,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/91bf2699f6204fc0b8da42c376cfe320 2023-04-07 09:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:52,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:52,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:52,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:52,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1235486e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:52,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,861 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:52,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:52,864 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861292858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861292858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861292860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861292867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:52,876 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:52,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:52,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:52,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:52,877 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:52,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:52,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,884 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,894 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,894 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,894 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:52,894 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:52,894 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:52,895 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:52,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:52,895 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:52,895 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:52,895 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:52,895 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:52,895 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:52,895 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:52,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:52,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/4b4ec9799e0d42bf8ffbe4c921b4c13d 2023-04-07 09:53:52,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/eef9f380433d4ea0ba3807fdced6c514 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/eef9f380433d4ea0ba3807fdced6c514 2023-04-07 09:53:52,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/eef9f380433d4ea0ba3807fdced6c514, entries=150, sequenceid=311, filesize=11.9 K 2023-04-07 09:53:52,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/91bf2699f6204fc0b8da42c376cfe320 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91bf2699f6204fc0b8da42c376cfe320 2023-04-07 09:53:52,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91bf2699f6204fc0b8da42c376cfe320, entries=150, sequenceid=311, filesize=11.9 K 2023-04-07 09:53:52,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/4b4ec9799e0d42bf8ffbe4c921b4c13d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4b4ec9799e0d42bf8ffbe4c921b4c13d 2023-04-07 09:53:52,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4b4ec9799e0d42bf8ffbe4c921b4c13d, entries=150, sequenceid=311, filesize=11.9 K 2023-04-07 09:53:52,996 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:52,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:52,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 0b39eaaa997d54d62464302ce4e71f9b in 666ms, sequenceid=311, compaction requested=true 2023-04-07 09:53:52,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:52,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:52,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:52,997 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:52,997 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:52,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:52,999 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:52,999 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:52,999 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,000 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/d702393a707a40b895bb2bc47403d9ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ea9df95c335a4b52b2f95bf970d85e0d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/eef9f380433d4ea0ba3807fdced6c514] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.1 K 2023-04-07 09:53:53,000 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:53,000 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:53,000 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,000 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0896455ab8f842e99efa8ac5af7d3dc2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1216b87b20ee4c21b34f2f92c94255e4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91bf2699f6204fc0b8da42c376cfe320] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.1 K 2023-04-07 09:53:53,001 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d702393a707a40b895bb2bc47403d9ed, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861231223 2023-04-07 09:53:53,001 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 0896455ab8f842e99efa8ac5af7d3dc2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861231223 2023-04-07 09:53:53,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:53,015 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1216b87b20ee4c21b34f2f92c94255e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680861231562 2023-04-07 09:53:53,015 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ea9df95c335a4b52b2f95bf970d85e0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680861231562 2023-04-07 09:53:53,016 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 91bf2699f6204fc0b8da42c376cfe320, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680861232212 2023-04-07 09:53:53,016 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting eef9f380433d4ea0ba3807fdced6c514, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680861232212 2023-04-07 09:53:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:53,017 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:53,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:53,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:53,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:53,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,021 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:53,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:53,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:53,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:53,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:53,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:53,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:53,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:53,037 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#42 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:53,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:53,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:53,039 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=154.31 KB heapSize=428.16 KB 2023-04-07 09:53:53,048 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:53,048 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:53,048 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:53,048 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:53,048 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:53,048 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:53,049 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:53,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:53,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:53,093 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/65e9686807d74837bb95118daa3b8c91 2023-04-07 09:53:53,097 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/96f4952e59c84c198800c769fc440825 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/96f4952e59c84c198800c769fc440825 2023-04-07 09:53:53,101 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/23b8a1d806e949dc8d5c0a6983b12d54 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/23b8a1d806e949dc8d5c0a6983b12d54 2023-04-07 09:53:53,112 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into 96f4952e59c84c198800c769fc440825(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:53,112 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:53,112 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=13, startTime=1680861232997; duration=0sec 2023-04-07 09:53:53,112 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 23b8a1d806e949dc8d5c0a6983b12d54(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:53,113 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:53,113 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=13, startTime=1680861232997; duration=0sec 2023-04-07 09:53:53,113 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:53,113 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:53,115 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:53,115 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:53,115 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,113 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:53,115 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bb68bc95db2f4dde88ebfee99f1822b9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ad6ffcab34649c9bf9254d4a93c2a60, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4b4ec9799e0d42bf8ffbe4c921b4c13d] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.1 K 2023-04-07 09:53:53,116 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bb68bc95db2f4dde88ebfee99f1822b9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861231223 2023-04-07 09:53:53,116 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5ad6ffcab34649c9bf9254d4a93c2a60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680861231562 2023-04-07 09:53:53,117 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4b4ec9799e0d42bf8ffbe4c921b4c13d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680861232212 2023-04-07 09:53:53,120 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/880addf3e2184be0869a89278efa530c 2023-04-07 09:53:53,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861293115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,149 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:53,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861293170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861293172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861293172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57306 deadline: 1680861293174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57366 deadline: 1680861293232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861293434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,574 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/60076e6079c24287b9de4f593e568288 2023-04-07 09:53:53,575 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/ce965889047f4d689b963337f4a7dcb3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ce965889047f4d689b963337f4a7dcb3 2023-04-07 09:53:53,587 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/65e9686807d74837bb95118daa3b8c91 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/65e9686807d74837bb95118daa3b8c91 2023-04-07 09:53:53,593 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into ce965889047f4d689b963337f4a7dcb3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:53,593 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:53,593 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=13, startTime=1680861232997; duration=0sec 2023-04-07 09:53:53,593 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:53,599 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/65e9686807d74837bb95118daa3b8c91, entries=150, sequenceid=400, filesize=11.9 K 2023-04-07 09:53:53,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/880addf3e2184be0869a89278efa530c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/880addf3e2184be0869a89278efa530c 2023-04-07 09:53:53,616 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/880addf3e2184be0869a89278efa530c, entries=150, sequenceid=400, filesize=11.9 K 2023-04-07 09:53:53,618 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/60076e6079c24287b9de4f593e568288 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/60076e6079c24287b9de4f593e568288 2023-04-07 09:53:53,625 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/60076e6079c24287b9de4f593e568288, entries=150, sequenceid=400, filesize=11.9 K 2023-04-07 09:53:53,626 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~428.11 KB/438384, currentSize=33.54 KB/34350 for 0b39eaaa997d54d62464302ce4e71f9b in 588ms, sequenceid=400, compaction requested=false 2023-04-07 09:53:53,626 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:53,626 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:53,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:53,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4135c668[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:53,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:53,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,634 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:53,634 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,635 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:53,636 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:53,636 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:53,636 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:53,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:53,639 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:53,639 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:53,639 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:53,639 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:53,639 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:53,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:53,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:53,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:53,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:53,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:53,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:53,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:53,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:53,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:53,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:53,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,657 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,657 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,657 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:53,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:53,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:53:53,677 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:53,677 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:53,677 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:53,677 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:53,677 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:53,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:53,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/06bf4464fa2748bf81b8dc455b2b295a 2023-04-07 09:53:53,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861233694, max timestamp=1680861233704Num uniques 859; 2023-04-07 09:53:53,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:53:53,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861233694, max timestamp=1680861233704Num uniques 859; 2023-04-07 09:53:53,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:53:53,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861233694, max timestamp=1680861233704Num uniques 859; 2023-04-07 09:53:53,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:53:53,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861293728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861293730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861293735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861293736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861293738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/cd11eff0026a4932a639452f241675bb 2023-04-07 09:53:53,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:53,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:53,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:53,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:53,780 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:53,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:53,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:53,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:53,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,782 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:53,782 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:53,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:53,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:53,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:53,797 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,797 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:53,797 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:53,797 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:53,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:53,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/4f0700d488f849468c8ca35572919c78 2023-04-07 09:53:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:53,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/06bf4464fa2748bf81b8dc455b2b295a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/06bf4464fa2748bf81b8dc455b2b295a 2023-04-07 09:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482ee723[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:53,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:53,827 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,828 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:53,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:53,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:53,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:53,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/06bf4464fa2748bf81b8dc455b2b295a, entries=200, sequenceid=413, filesize=14.3 K 2023-04-07 09:53:53,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:53,835 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:53,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:53,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:53,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:53,835 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:53,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/cd11eff0026a4932a639452f241675bb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/cd11eff0026a4932a639452f241675bb 2023-04-07 09:53:53,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:53,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:53,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:53,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:53,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861293838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861293838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861293839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:53,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861293841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/cd11eff0026a4932a639452f241675bb, entries=150, sequenceid=413, filesize=11.9 K 2023-04-07 09:53:53,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:53,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:53,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:53,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:53,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:53,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:53,849 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:53,850 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:53,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:53,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:53,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:53,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/4f0700d488f849468c8ca35572919c78 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4f0700d488f849468c8ca35572919c78 2023-04-07 09:53:53,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4f0700d488f849468c8ca35572919c78, entries=150, sequenceid=413, filesize=11.9 K 2023-04-07 09:53:53,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 0b39eaaa997d54d62464302ce4e71f9b in 181ms, sequenceid=413, compaction requested=true 2023-04-07 09:53:53,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:53,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:53,859 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:53,859 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:53,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:53,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39476 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:53,860 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:53,860 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,860 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,860 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/96f4952e59c84c198800c769fc440825, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/65e9686807d74837bb95118daa3b8c91, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/06bf4464fa2748bf81b8dc455b2b295a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=38.6 K 2023-04-07 09:53:53,860 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/23b8a1d806e949dc8d5c0a6983b12d54, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/880addf3e2184be0869a89278efa530c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/cd11eff0026a4932a639452f241675bb] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.2 K 2023-04-07 09:53:53,861 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 96f4952e59c84c198800c769fc440825, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680861232212 2023-04-07 09:53:53,862 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 23b8a1d806e949dc8d5c0a6983b12d54, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680861232212 2023-04-07 09:53:53,863 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 65e9686807d74837bb95118daa3b8c91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680861232530 2023-04-07 09:53:53,863 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 880addf3e2184be0869a89278efa530c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680861232530 2023-04-07 09:53:53,864 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 06bf4464fa2748bf81b8dc455b2b295a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680861233099 2023-04-07 09:53:53,864 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cd11eff0026a4932a639452f241675bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680861233099 2023-04-07 09:53:53,873 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#51 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:53,875 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#52 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:53,904 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/ff8a562ab2574302b2202ab4fe5fe85c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/ff8a562ab2574302b2202ab4fe5fe85c 2023-04-07 09:53:53,912 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into ff8a562ab2574302b2202ab4fe5fe85c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:53,912 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:53,912 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=13, startTime=1680861233859; duration=0sec 2023-04-07 09:53:53,912 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:53,912 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:53:53,914 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:53:53,914 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:53,914 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,914 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ce965889047f4d689b963337f4a7dcb3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/60076e6079c24287b9de4f593e568288, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4f0700d488f849468c8ca35572919c78] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.2 K 2023-04-07 09:53:53,915 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ce965889047f4d689b963337f4a7dcb3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680861232212 2023-04-07 09:53:53,915 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 60076e6079c24287b9de4f593e568288, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680861232530 2023-04-07 09:53:53,916 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4f0700d488f849468c8ca35572919c78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680861233099 2023-04-07 09:53:53,926 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#53 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:53,944 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/b51900de50944e6db09789419c19117f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b51900de50944e6db09789419c19117f 2023-04-07 09:53:53,950 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:53,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:53,954 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into b51900de50944e6db09789419c19117f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:53,954 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:53,954 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=13, startTime=1680861233859; duration=0sec 2023-04-07 09:53:53,954 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:53,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:53,956 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:53,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:53,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:53,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:53,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,959 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:53,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:53,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:53,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:53,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:53,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:53,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:53,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:53,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:53,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:53,962 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:53,962 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:53,962 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-07 09:53:53,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:53,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:53,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:53,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:53,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:53,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:54,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:54,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:54,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861294052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861294054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861294055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861294055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861294156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861294160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861294160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861294160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861294239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,305 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/525916fa01aa4f90bf0bddb35b399b1f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/525916fa01aa4f90bf0bddb35b399b1f 2023-04-07 09:53:54,313 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into 525916fa01aa4f90bf0bddb35b399b1f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:54,314 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:54,314 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=13, startTime=1680861233859; duration=0sec 2023-04-07 09:53:54,314 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:54,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861294358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861294365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861294366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861294365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,380 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/76872dbd8e00435a9223a7b3f5d1ac06 2023-04-07 09:53:54,394 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/bd74e6bd25f544eb8ff1243b520aeffe 2023-04-07 09:53:54,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861294662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861294672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861294672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861294675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,816 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/49bd425545e5463093f302f7fd1ece89 2023-04-07 09:53:54,824 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/76872dbd8e00435a9223a7b3f5d1ac06 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/76872dbd8e00435a9223a7b3f5d1ac06 2023-04-07 09:53:54,841 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/76872dbd8e00435a9223a7b3f5d1ac06, entries=150, sequenceid=450, filesize=11.9 K 2023-04-07 09:53:54,842 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/bd74e6bd25f544eb8ff1243b520aeffe as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/bd74e6bd25f544eb8ff1243b520aeffe 2023-04-07 09:53:54,850 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/bd74e6bd25f544eb8ff1243b520aeffe, entries=150, sequenceid=450, filesize=11.9 K 2023-04-07 09:53:54,851 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/49bd425545e5463093f302f7fd1ece89 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/49bd425545e5463093f302f7fd1ece89 2023-04-07 09:53:54,863 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/49bd425545e5463093f302f7fd1ece89, entries=150, sequenceid=450, filesize=11.9 K 2023-04-07 09:53:54,865 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 0b39eaaa997d54d62464302ce4e71f9b in 903ms, sequenceid=450, compaction requested=false 2023-04-07 09:53:54,865 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:54,866 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:54,881 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67efb26b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:54,885 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:54,885 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,887 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:54,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:54,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:54,888 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,888 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:54,890 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:54,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:54,896 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:54,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:54,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:54,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:54,896 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:54,898 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:54,898 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:54,898 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:54,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:54,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:54,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:54,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:54,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:54,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:54,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:54,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:54,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:54,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:54,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:54,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:54,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:54,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:54,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:54,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:54,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:54,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:54,905 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:54,905 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:54,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:54,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:55,005 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:55,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:55,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:55,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:55,010 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:55,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:55,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:55,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:55,012 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:55,013 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,013 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:55,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:55,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:55,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:55,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:55,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:55,015 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,015 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:55,015 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:53:55,015 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:55,015 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:55,015 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:55,016 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:55,016 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:55,016 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:55,035 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/683a1d55c1144f4c87acc703d4059479 2023-04-07 09:53:55,062 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/c15c12f9ef6a43718c97c1ac3ea0c950 2023-04-07 09:53:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:55,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:55,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680861235187, max timestamp=1680861235202Num uniques 859; 2023-04-07 09:53:55,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680861235187, max timestamp=1680861235202Num uniques 859; 2023-04-07 09:53:55,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-07 09:53:55,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-07 09:53:55,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861235191, max timestamp=1680861235202Num uniques 859; 2023-04-07 09:53:55,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:53:55,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861295268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861295273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861295277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861295275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861295277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861295380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861295384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861295385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861295386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861295386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,507 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/917decd7ffbb46b5ade856405e40c7c3 2023-04-07 09:53:55,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/683a1d55c1144f4c87acc703d4059479 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/683a1d55c1144f4c87acc703d4059479 2023-04-07 09:53:55,522 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/683a1d55c1144f4c87acc703d4059479, entries=150, sequenceid=461, filesize=11.9 K 2023-04-07 09:53:55,523 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/c15c12f9ef6a43718c97c1ac3ea0c950 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c15c12f9ef6a43718c97c1ac3ea0c950 2023-04-07 09:53:55,532 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c15c12f9ef6a43718c97c1ac3ea0c950, entries=150, sequenceid=461, filesize=11.9 K 2023-04-07 09:53:55,534 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/917decd7ffbb46b5ade856405e40c7c3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/917decd7ffbb46b5ade856405e40c7c3 2023-04-07 09:53:55,541 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/917decd7ffbb46b5ade856405e40c7c3, entries=150, sequenceid=461, filesize=11.9 K 2023-04-07 09:53:55,543 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=143.13 KB/146560 for 0b39eaaa997d54d62464302ce4e71f9b in 528ms, sequenceid=461, compaction requested=true 2023-04-07 09:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:55,543 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:55,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:55,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:55,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa1f0c0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:55,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:55,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:55,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:55,554 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:55,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:55,558 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:55,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:55,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:55,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:55,558 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:55,560 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,560 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:55,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,560 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:55,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:55,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:55,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:55,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:55,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:55,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:55,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:55,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,572 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:55,572 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:55,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:55,587 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=149.83 KB heapSize=415.84 KB 2023-04-07 09:53:55,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:55,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:55,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:55,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:55,587 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:55,587 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:55,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861295595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861295596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861295596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861295597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861295597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/63d57f4b4fc142c5b9f276ebea520d4c 2023-04-07 09:53:55,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/a3915e9e512b4a50994bcfad667b03c1 2023-04-07 09:53:55,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/b0aab677a4e04a8ab5d29960664809de 2023-04-07 09:53:55,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/63d57f4b4fc142c5b9f276ebea520d4c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d57f4b4fc142c5b9f276ebea520d4c 2023-04-07 09:53:55,672 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:55,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:55,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d57f4b4fc142c5b9f276ebea520d4c, entries=200, sequenceid=497, filesize=14.3 K 2023-04-07 09:53:55,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/a3915e9e512b4a50994bcfad667b03c1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a3915e9e512b4a50994bcfad667b03c1 2023-04-07 09:53:55,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:55,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:55,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:55,677 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:55,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:55,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:55,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,679 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:55,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:55,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:55,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:55,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:55,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:55,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a3915e9e512b4a50994bcfad667b03c1, entries=150, sequenceid=497, filesize=11.9 K 2023-04-07 09:53:55,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/b0aab677a4e04a8ab5d29960664809de as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b0aab677a4e04a8ab5d29960664809de 2023-04-07 09:53:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:55,686 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:55,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30487bda[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:55,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:55,689 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b0aab677a4e04a8ab5d29960664809de, entries=150, sequenceid=497, filesize=11.9 K 2023-04-07 09:53:55,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~415.80 KB/425776, currentSize=40.25 KB/41220 for 0b39eaaa997d54d62464302ce4e71f9b in 104ms, sequenceid=497, compaction requested=true 2023-04-07 09:53:55,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:55,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:55,690 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:55,690 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:55,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:55,692 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:55,692 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51764 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:55,693 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:55,693 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:55,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:55,693 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,693 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,693 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,693 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/ff8a562ab2574302b2202ab4fe5fe85c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/bd74e6bd25f544eb8ff1243b520aeffe, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c15c12f9ef6a43718c97c1ac3ea0c950, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a3915e9e512b4a50994bcfad667b03c1] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=48.2 K 2023-04-07 09:53:55,693 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/525916fa01aa4f90bf0bddb35b399b1f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/76872dbd8e00435a9223a7b3f5d1ac06, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/683a1d55c1144f4c87acc703d4059479, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d57f4b4fc142c5b9f276ebea520d4c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=50.6 K 2023-04-07 09:53:55,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:55,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:55,694 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ff8a562ab2574302b2202ab4fe5fe85c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680861233099 2023-04-07 09:53:55,694 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 525916fa01aa4f90bf0bddb35b399b1f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680861233099 2023-04-07 09:53:55,694 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting bd74e6bd25f544eb8ff1243b520aeffe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680861233723 2023-04-07 09:53:55,694 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 76872dbd8e00435a9223a7b3f5d1ac06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680861233723 2023-04-07 09:53:55,695 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c15c12f9ef6a43718c97c1ac3ea0c950, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680861234050 2023-04-07 09:53:55,695 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 683a1d55c1144f4c87acc703d4059479, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680861234050 2023-04-07 09:53:55,695 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a3915e9e512b4a50994bcfad667b03c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680861235217 2023-04-07 09:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:55,695 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 63d57f4b4fc142c5b9f276ebea520d4c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680861235217 2023-04-07 09:53:55,696 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,704 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:55,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:55,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:55,705 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:55,705 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:55,705 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:55,709 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:55,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:55,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:55,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:55,709 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:55,711 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,711 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:55,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,711 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,715 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:55,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:55,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:55,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:55,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:55,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:55,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:55,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,720 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:55,721 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#65 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:55,721 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:55,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:55,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861235724, max timestamp=1680861235731Num uniques 859; 2023-04-07 09:53:55,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:53:55,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861235724, max timestamp=1680861235731Num uniques 859; 2023-04-07 09:53:55,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:53:55,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861235724, max timestamp=1680861235731Num uniques 859; 2023-04-07 09:53:55,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:53:55,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/b3e088279bba437a9dfa6789958e2d4d 2023-04-07 09:53:55,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/f30038c3ee274591a37ac12ae91a7c6c 2023-04-07 09:53:55,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861295747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861295748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861295748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861295749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861295755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,821 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:55,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:55,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:55,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:55,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:55,825 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:55,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:55,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:55,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,828 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:55,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:55,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:55,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:55,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:55,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:55,848 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,848 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:55,848 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:55,848 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:55,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:55,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f60850f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:55,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:55,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:55,860 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,860 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:55,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861295858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861295858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:55,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:55,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:55,864 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:55,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:55,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:55,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:55,865 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:55,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861295859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861295859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,868 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:55,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861295859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,870 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:55,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:55,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:55,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:55,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:55,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:55,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:55,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:55,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:55,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:55,975 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:55,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:55,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:55,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:55,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:55,980 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:55,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:55,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:55,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:55,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:55,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:55,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:55,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:55,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:55,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:55,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:55,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:55,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:55,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:55,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15060262[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:55,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:55,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,993 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:55,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,996 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:55,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:56,000 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:56,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:56,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:56,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:56,000 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:56,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,003 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:56,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:56,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:56,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:56,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:56,012 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:56,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:56,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861296062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861296063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861296068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861296069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861296069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:56,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:56,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:56,116 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:56,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:56,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:56,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:56,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:56,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:56,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:56,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:56,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:56,126 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:56,126 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:56,126 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:56,127 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:56,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:56,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:56,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4571c7f2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:56,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:56,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,137 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:56,137 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:56,137 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:56,139 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,139 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:56,140 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/1fd3ea69a21e49bd8c4cb27de38773fd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1fd3ea69a21e49bd8c4cb27de38773fd 2023-04-07 09:53:56,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:56,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:56,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:56,145 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,148 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/5ad1b6c28ade4cb5bfc775c61c9ec789 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5ad1b6c28ade4cb5bfc775c61c9ec789 2023-04-07 09:53:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:56,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:56,151 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:56,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:56,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:56,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:56,151 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:56,152 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 1fd3ea69a21e49bd8c4cb27de38773fd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:56,152 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:56,152 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=12, startTime=1680861235690; duration=0sec 2023-04-07 09:53:56,152 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:56,152 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:56,153 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,153 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,153 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,154 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:56,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,157 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into 5ad1b6c28ade4cb5bfc775c61c9ec789(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:56,157 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,158 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=12, startTime=1680861235690; duration=0sec 2023-04-07 09:53:56,158 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,158 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,158 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,159 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:56,159 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:56,159 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b51900de50944e6db09789419c19117f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/49bd425545e5463093f302f7fd1ece89, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/917decd7ffbb46b5ade856405e40c7c3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b0aab677a4e04a8ab5d29960664809de] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=48.2 K 2023-04-07 09:53:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,159 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b51900de50944e6db09789419c19117f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1680861233099 2023-04-07 09:53:56,160 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 49bd425545e5463093f302f7fd1ece89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680861233723 2023-04-07 09:53:56,160 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:56,161 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:56,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:56,161 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 917decd7ffbb46b5ade856405e40c7c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680861234050 2023-04-07 09:53:56,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:56,161 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b0aab677a4e04a8ab5d29960664809de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680861235217 2023-04-07 09:53:56,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:56,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:56,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:56,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,163 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,163 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,163 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:56,163 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:56,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:56,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/4416dda0d1b744f68aa06c1122647d00 2023-04-07 09:53:56,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/b3e088279bba437a9dfa6789958e2d4d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/b3e088279bba437a9dfa6789958e2d4d 2023-04-07 09:53:56,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/b3e088279bba437a9dfa6789958e2d4d, entries=150, sequenceid=508, filesize=11.9 K 2023-04-07 09:53:56,190 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:56,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/f30038c3ee274591a37ac12ae91a7c6c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/f30038c3ee274591a37ac12ae91a7c6c 2023-04-07 09:53:56,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/f30038c3ee274591a37ac12ae91a7c6c, entries=150, sequenceid=508, filesize=11.9 K 2023-04-07 09:53:56,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/4416dda0d1b744f68aa06c1122647d00 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4416dda0d1b744f68aa06c1122647d00 2023-04-07 09:53:56,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4416dda0d1b744f68aa06c1122647d00, entries=150, sequenceid=508, filesize=11.9 K 2023-04-07 09:53:56,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.68 KB/129720 for 0b39eaaa997d54d62464302ce4e71f9b in 515ms, sequenceid=508, compaction requested=false 2023-04-07 09:53:56,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:56,227 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/ed30df31f73b40058d5581dd7ddb21b2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ed30df31f73b40058d5581dd7ddb21b2 2023-04-07 09:53:56,233 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into ed30df31f73b40058d5581dd7ddb21b2(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:56,234 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:56,234 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=12, startTime=1680861235693; duration=0sec 2023-04-07 09:53:56,234 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:56,263 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:56,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:56,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:56,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:56,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:56,267 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:56,267 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:56,267 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:56,267 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:56,267 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,267 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,269 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:56,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:56,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:56,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:56,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:56,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:56,271 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:56,271 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:56,271 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=126.68 KB heapSize=363.23 KB 2023-04-07 09:53:56,271 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:56,271 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:56,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:56,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:56,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:56,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:56,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:56,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861296377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861296379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861296385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861296385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861296385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861296487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861296492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861296492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861296493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,682 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/99416ef10aaa42cb9c4b857cba2258f9 2023-04-07 09:53:56,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861296690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861296697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861296698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861296700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,716 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/7e7b912f8fc0470bbd4a68acc2a546df 2023-04-07 09:53:56,757 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/8d523b5835ff41cf8a9de904fcfe3ea2 2023-04-07 09:53:56,765 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/99416ef10aaa42cb9c4b857cba2258f9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/99416ef10aaa42cb9c4b857cba2258f9 2023-04-07 09:53:56,773 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/99416ef10aaa42cb9c4b857cba2258f9, entries=150, sequenceid=547, filesize=11.9 K 2023-04-07 09:53:56,774 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/7e7b912f8fc0470bbd4a68acc2a546df as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7e7b912f8fc0470bbd4a68acc2a546df 2023-04-07 09:53:56,780 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7e7b912f8fc0470bbd4a68acc2a546df, entries=150, sequenceid=547, filesize=11.9 K 2023-04-07 09:53:56,781 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/8d523b5835ff41cf8a9de904fcfe3ea2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/8d523b5835ff41cf8a9de904fcfe3ea2 2023-04-07 09:53:56,788 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/8d523b5835ff41cf8a9de904fcfe3ea2, entries=150, sequenceid=547, filesize=11.9 K 2023-04-07 09:53:56,789 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.68 KB/129720, heapSize ~363.19 KB/371904, currentSize=60.38 KB/61830 for 0b39eaaa997d54d62464302ce4e71f9b in 518ms, sequenceid=547, compaction requested=true 2023-04-07 09:53:56,789 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:56,789 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:56,789 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:56,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:56,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:56,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3614e22d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:56,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:56,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,798 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:56,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:56,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:56,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:56,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:56,808 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:56,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:56,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:56,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:56,808 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:56,810 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,810 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:56,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,812 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:56,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:56,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:56,819 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:56,819 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:56,819 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:56,819 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:56,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:56,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:56,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:56,889 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:53:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:56,890 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:56,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/e3103480eb2f4eebb9ab21cdb6a590ba 2023-04-07 09:53:56,919 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:56,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:56,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:56,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:56,938 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:56,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:56,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:56,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:56,939 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,939 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,941 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:56,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:56,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:56,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:56,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:56,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861296951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:56,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:56,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:56,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:56,956 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/aaf3a0253803449bbe8b8af7cfe07285 2023-04-07 09:53:56,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:56,973 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:56,974 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:56,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:56,974 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:56,976 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:56,976 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:56,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:56,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd7511d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:56,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:56,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:56,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:56,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:56,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:56,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:56,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/06094dff34044f54b0a49939026893b8 2023-04-07 09:53:56,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:56,992 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:56,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:56,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:56,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:56,992 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:56,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/e3103480eb2f4eebb9ab21cdb6a590ba as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e3103480eb2f4eebb9ab21cdb6a590ba 2023-04-07 09:53:56,994 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,994 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:56,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,994 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:56,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,998 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861296996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e3103480eb2f4eebb9ab21cdb6a590ba, entries=200, sequenceid=560, filesize=14.3 K 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/aaf3a0253803449bbe8b8af7cfe07285 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/aaf3a0253803449bbe8b8af7cfe07285 2023-04-07 09:53:57,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,003 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:57,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:57,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:57,003 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:57,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:57,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861297003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861297005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861297002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/aaf3a0253803449bbe8b8af7cfe07285, entries=150, sequenceid=560, filesize=11.9 K 2023-04-07 09:53:57,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/06094dff34044f54b0a49939026893b8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/06094dff34044f54b0a49939026893b8 2023-04-07 09:53:57,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/06094dff34044f54b0a49939026893b8, entries=150, sequenceid=560, filesize=11.9 K 2023-04-07 09:53:57,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0b39eaaa997d54d62464302ce4e71f9b in 136ms, sequenceid=560, compaction requested=true 2023-04-07 09:53:57,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:57,025 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:57,025 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:57,028 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:57,028 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:57,028 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:57,028 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:57,029 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,029 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,029 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1fd3ea69a21e49bd8c4cb27de38773fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/f30038c3ee274591a37ac12ae91a7c6c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7e7b912f8fc0470bbd4a68acc2a546df, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/aaf3a0253803449bbe8b8af7cfe07285] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=48.3 K 2023-04-07 09:53:57,029 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5ad1b6c28ade4cb5bfc775c61c9ec789, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/b3e088279bba437a9dfa6789958e2d4d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/99416ef10aaa42cb9c4b857cba2258f9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e3103480eb2f4eebb9ab21cdb6a590ba] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=50.7 K 2023-04-07 09:53:57,031 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1fd3ea69a21e49bd8c4cb27de38773fd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680861235217 2023-04-07 09:53:57,031 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5ad1b6c28ade4cb5bfc775c61c9ec789, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680861235217 2023-04-07 09:53:57,031 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f30038c3ee274591a37ac12ae91a7c6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861235595 2023-04-07 09:53:57,031 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b3e088279bba437a9dfa6789958e2d4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861235595 2023-04-07 09:53:57,032 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7e7b912f8fc0470bbd4a68acc2a546df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680861235747 2023-04-07 09:53:57,032 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 99416ef10aaa42cb9c4b857cba2258f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680861235747 2023-04-07 09:53:57,032 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting aaf3a0253803449bbe8b8af7cfe07285, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680861236379 2023-04-07 09:53:57,032 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e3103480eb2f4eebb9ab21cdb6a590ba, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680861236374 2023-04-07 09:53:57,054 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#75 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:57,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:57,059 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:53:57,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:57,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:57,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:57,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:57,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:57,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:57,065 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:57,103 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:57,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:57,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:57,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861297110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:57,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:57,131 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:57,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:57,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:57,133 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,133 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,135 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:57,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:57,151 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,151 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:57,151 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:57,151 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:57,154 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186741d7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:57,156 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:57,156 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,158 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,158 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:57,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:57,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:57,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:57,160 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:57,163 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:57,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:57,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:57,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:57,163 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:57,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:57,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:57,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,179 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:57,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:57,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861297215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:57,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:57,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:57,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:57,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:57,285 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:57,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:57,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:57,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,287 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,288 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,288 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:57,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:57,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:57,289 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,289 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:57,290 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:57,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53625b7e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:57,294 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:57,294 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,295 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:57,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:57,299 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:57,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:57,302 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:57,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:57,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:57,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:57,302 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:57,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:57,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,308 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:57,308 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:57,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:57,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:57,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:57,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:57,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:57,415 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:57,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:57,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:57,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,418 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,419 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,419 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:57,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861297417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:57,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:57,431 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f6deb5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:57,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:57,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,437 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:57,437 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,437 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:57,440 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:57,444 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:57,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:57,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:57,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:57,445 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:57,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,450 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:57,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:57,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,454 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:57,454 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:57,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861297503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,505 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/5b282fdb80b54968adc4aa9d071d039d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/5b282fdb80b54968adc4aa9d071d039d 2023-04-07 09:53:57,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861297509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861297509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,512 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 5b282fdb80b54968adc4aa9d071d039d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:57,512 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:57,512 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=12, startTime=1680861237025; duration=0sec 2023-04-07 09:53:57,513 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:57,513 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:53:57,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/e42f1d25289149b8b3f7d8ebca1fdbd0 2023-04-07 09:53:57,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861297511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,515 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:53:57,515 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:57,515 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,515 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ed30df31f73b40058d5581dd7ddb21b2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4416dda0d1b744f68aa06c1122647d00, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/8d523b5835ff41cf8a9de904fcfe3ea2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/06094dff34044f54b0a49939026893b8] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=48.3 K 2023-04-07 09:53:57,516 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ed30df31f73b40058d5581dd7ddb21b2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680861235217 2023-04-07 09:53:57,516 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4416dda0d1b744f68aa06c1122647d00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861235595 2023-04-07 09:53:57,517 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8d523b5835ff41cf8a9de904fcfe3ea2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1680861235747 2023-04-07 09:53:57,517 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 06094dff34044f54b0a49939026893b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680861236379 2023-04-07 09:53:57,529 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:57,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/6ec1134433b2409f869be3466222f6f4 2023-04-07 09:53:57,544 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/e98235fd952c4de0a5fabf5d3a375d28 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/e98235fd952c4de0a5fabf5d3a375d28 2023-04-07 09:53:57,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/1adc8b17adb3480aa2fba584413eb4c1 2023-04-07 09:53:57,553 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/f102afb9d9664a59b30b2e360b5375f2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f102afb9d9664a59b30b2e360b5375f2 2023-04-07 09:53:57,555 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:57,555 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into e98235fd952c4de0a5fabf5d3a375d28(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:57,555 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:57,555 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=12, startTime=1680861237025; duration=0sec 2023-04-07 09:53:57,555 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:57,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/e42f1d25289149b8b3f7d8ebca1fdbd0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e42f1d25289149b8b3f7d8ebca1fdbd0 2023-04-07 09:53:57,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:57,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:57,560 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:57,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:57,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:57,561 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into f102afb9d9664a59b30b2e360b5375f2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:57,561 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:57,561 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=12, startTime=1680861237025; duration=0sec 2023-04-07 09:53:57,561 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:57,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e42f1d25289149b8b3f7d8ebca1fdbd0, entries=200, sequenceid=580, filesize=14.3 K 2023-04-07 09:53:57,563 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/6ec1134433b2409f869be3466222f6f4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/6ec1134433b2409f869be3466222f6f4 2023-04-07 09:53:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:57,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:57,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:57,565 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,565 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:57,565 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:57,566 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:57,566 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:57,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/6ec1134433b2409f869be3466222f6f4, entries=150, sequenceid=580, filesize=11.9 K 2023-04-07 09:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:57,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/1adc8b17adb3480aa2fba584413eb4c1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1adc8b17adb3480aa2fba584413eb4c1 2023-04-07 09:53:57,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ae204f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:57,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:57,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:57,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:57,575 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:57,578 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:57,578 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:57,578 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:57,578 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:57,578 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:57,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:57,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,582 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,582 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1adc8b17adb3480aa2fba584413eb4c1, entries=150, sequenceid=580, filesize=11.9 K 2023-04-07 09:53:57,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0b39eaaa997d54d62464302ce4e71f9b in 529ms, sequenceid=580, compaction requested=false 2023-04-07 09:53:57,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:57,588 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,588 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:57,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:57,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:57,589 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:57,589 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:57,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:57,588 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,590 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,590 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,590 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,590 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,590 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:57,590 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:57,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:57,689 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:57,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:57,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:57,692 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:57,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:57,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:57,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:57,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:57,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:57,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:57,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:57,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:57,697 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:57,698 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:57,698 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:53:57,698 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:57,698 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:57,698 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:57,698 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:57,698 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:57,698 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:57,707 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/82b357cf70e04db485f23a103977d1fa 2023-04-07 09:53:57,722 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/a101ae942b2d4d398b97ef9ff1038375 2023-04-07 09:53:57,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:57,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861297778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:57,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:57,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861297882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861298086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,137 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/5ae263c51ab74c1788c2488b70225c77 2023-04-07 09:53:58,142 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/82b357cf70e04db485f23a103977d1fa as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/82b357cf70e04db485f23a103977d1fa 2023-04-07 09:53:58,147 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/82b357cf70e04db485f23a103977d1fa, entries=150, sequenceid=595, filesize=11.9 K 2023-04-07 09:53:58,148 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/a101ae942b2d4d398b97ef9ff1038375 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a101ae942b2d4d398b97ef9ff1038375 2023-04-07 09:53:58,153 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a101ae942b2d4d398b97ef9ff1038375, entries=150, sequenceid=595, filesize=11.9 K 2023-04-07 09:53:58,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/5ae263c51ab74c1788c2488b70225c77 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ae263c51ab74c1788c2488b70225c77 2023-04-07 09:53:58,159 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ae263c51ab74c1788c2488b70225c77, entries=150, sequenceid=595, filesize=11.9 K 2023-04-07 09:53:58,161 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0b39eaaa997d54d62464302ce4e71f9b in 463ms, sequenceid=595, compaction requested=true 2023-04-07 09:53:58,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:58,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,163 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:58,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67e5a8d4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:58,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:58,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:58,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:58,167 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:58,169 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:58,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:58,180 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:58,180 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:58,180 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:58,180 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:58,180 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:58,182 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,183 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,187 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:58,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:58,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:58,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,187 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:58,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,187 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:58,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,288 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:58,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:58,291 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:58,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:58,292 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,292 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,293 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,293 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,294 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,294 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:58,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:58,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:58,295 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,295 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:58,296 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:53:58,296 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:58,296 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:58,296 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:58,296 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:58,296 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:58,296 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:58,306 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/52b9f49bdb224f8289887b8213db8c29 2023-04-07 09:53:58,318 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/2d60e6b0e3a64cddaad922358e1e27da 2023-04-07 09:53:58,347 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/99d1563357af47188101e27610f877f0 2023-04-07 09:53:58,357 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/52b9f49bdb224f8289887b8213db8c29 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/52b9f49bdb224f8289887b8213db8c29 2023-04-07 09:53:58,363 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/52b9f49bdb224f8289887b8213db8c29, entries=150, sequenceid=615, filesize=11.9 K 2023-04-07 09:53:58,364 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/2d60e6b0e3a64cddaad922358e1e27da as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2d60e6b0e3a64cddaad922358e1e27da 2023-04-07 09:53:58,374 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2d60e6b0e3a64cddaad922358e1e27da, entries=150, sequenceid=615, filesize=11.9 K 2023-04-07 09:53:58,375 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/99d1563357af47188101e27610f877f0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/99d1563357af47188101e27610f877f0 2023-04-07 09:53:58,394 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/99d1563357af47188101e27610f877f0, entries=150, sequenceid=615, filesize=11.9 K 2023-04-07 09:53:58,396 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 0b39eaaa997d54d62464302ce4e71f9b in 100ms, sequenceid=615, compaction requested=true 2023-04-07 09:53:58,396 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:58,396 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:58,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:58,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@261da6fd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:58,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:58,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:58,404 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,404 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:58,405 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,406 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:58,406 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,406 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:58,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:58,410 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:58,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:58,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:58,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:58,410 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:58,411 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,411 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,411 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,412 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,418 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:58,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:58,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,418 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:58,419 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:58,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:58,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:53:58,420 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:58,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:58,421 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:53:58,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/63d1969904de4784b5257df5959ef291 2023-04-07 09:53:58,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/3218114cbce54659ae3c5f30eb6d368a 2023-04-07 09:53:58,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861238468, max timestamp=1680861238490Num uniques 859; 2023-04-07 09:53:58,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861238468, max timestamp=1680861238490Num uniques 859; 2023-04-07 09:53:58,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861238468, max timestamp=1680861238490Num uniques 859; 2023-04-07 09:53:58,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:53:58,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:53:58,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:53:58,519 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:58,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:58,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:58,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:58,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:58,528 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:58,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:58,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:58,529 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,529 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,530 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:58,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:58,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:58,535 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:58,535 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,536 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:58,536 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:58,536 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:58,538 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:58,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861298537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861298538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526ac7fd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:58,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:58,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861298540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861298540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,544 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,544 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:58,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:58,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861298529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,546 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:58,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:58,549 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:58,549 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:58,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,555 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,557 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:58,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:58,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,562 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:58,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,562 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:58,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861298643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861298643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861298643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861298644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861298645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,662 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:58,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:58,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:58,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:58,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:58,668 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:58,668 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,668 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:58,668 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:58,669 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,669 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:58,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:58,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:58,680 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,680 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:58,680 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:58,680 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@667229a7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:58,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:58,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,685 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:58,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:58,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:58,691 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:58,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:58,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:58,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:58,691 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:58,693 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,693 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,693 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:58,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:58,698 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,698 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,698 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:58,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:58,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:58,803 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:58,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:58,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:58,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,805 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,805 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:58,806 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:58,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:58,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:58,810 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,810 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:58,810 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:58,810 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:53:58,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:53:58,812 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:53:58,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1b8eb5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:53:58,814 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:53:58,814 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,815 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:53:58,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:53:58,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:53:58,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,817 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:53:58,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:53:58,823 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:53:58,823 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:53:58,823 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:53:58,823 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:53:58,823 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:53:58,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:53:58,827 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:53:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:53:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:53:58,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:53:58,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:53:58,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:53:58,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,832 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:53:58,832 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:53:58,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:53:58,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:53:58,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:53:58,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861298848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861298849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861298849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861298850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861298850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/66415c78c7c347c2af2f6b9747f40ac6 2023-04-07 09:53:58,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/63d1969904de4784b5257df5959ef291 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d1969904de4784b5257df5959ef291 2023-04-07 09:53:58,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d1969904de4784b5257df5959ef291, entries=200, sequenceid=625, filesize=14.3 K 2023-04-07 09:53:58,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/3218114cbce54659ae3c5f30eb6d368a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3218114cbce54659ae3c5f30eb6d368a 2023-04-07 09:53:58,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3218114cbce54659ae3c5f30eb6d368a, entries=150, sequenceid=625, filesize=11.9 K 2023-04-07 09:53:58,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/66415c78c7c347c2af2f6b9747f40ac6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/66415c78c7c347c2af2f6b9747f40ac6 2023-04-07 09:53:58,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/66415c78c7c347c2af2f6b9747f40ac6, entries=150, sequenceid=625, filesize=11.9 K 2023-04-07 09:53:58,928 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.15 KB/134300 for 0b39eaaa997d54d62464302ce4e71f9b in 508ms, sequenceid=625, compaction requested=true 2023-04-07 09:53:58,928 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:58,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:58,928 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:53:58,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:58,928 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:53:58,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:58,931 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66662 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:53:58,931 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:53:58,931 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,931 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f102afb9d9664a59b30b2e360b5375f2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e42f1d25289149b8b3f7d8ebca1fdbd0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/82b357cf70e04db485f23a103977d1fa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/52b9f49bdb224f8289887b8213db8c29, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d1969904de4784b5257df5959ef291] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=65.1 K 2023-04-07 09:53:58,932 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:53:58,932 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:53:58,932 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:53:58,932 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,932 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting f102afb9d9664a59b30b2e360b5375f2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680861236379 2023-04-07 09:53:58,932 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/5b282fdb80b54968adc4aa9d071d039d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/6ec1134433b2409f869be3466222f6f4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a101ae942b2d4d398b97ef9ff1038375, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2d60e6b0e3a64cddaad922358e1e27da, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3218114cbce54659ae3c5f30eb6d368a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=60.3 K 2023-04-07 09:53:58,933 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e42f1d25289149b8b3f7d8ebca1fdbd0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680861236942 2023-04-07 09:53:58,933 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 5b282fdb80b54968adc4aa9d071d039d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680861236379 2023-04-07 09:53:58,934 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 82b357cf70e04db485f23a103977d1fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680861237095 2023-04-07 09:53:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:53:58,935 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 6ec1134433b2409f869be3466222f6f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680861236947 2023-04-07 09:53:58,936 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 52b9f49bdb224f8289887b8213db8c29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680861237737 2023-04-07 09:53:58,936 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a101ae942b2d4d398b97ef9ff1038375, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680861237095 2023-04-07 09:53:58,936 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 63d1969904de4784b5257df5959ef291, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861238400 2023-04-07 09:53:58,937 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2d60e6b0e3a64cddaad922358e1e27da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680861237737 2023-04-07 09:53:58,938 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3218114cbce54659ae3c5f30eb6d368a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861238400 2023-04-07 09:53:58,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:53:58,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:53:58,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:53:58,939 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:53:58,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:53:58,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:53:58,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,942 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,942 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:53:58,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:53:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:53:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,943 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:58,943 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:53:58,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:53:58,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:53:58,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:53:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:53:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:53:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:53:58,956 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:58,956 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:53:58,956 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=131.15 KB heapSize=373.39 KB 2023-04-07 09:53:58,956 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:53:58,957 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:58,957 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:53:58,957 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:58,957 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:53:58,957 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:53:58,963 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#90 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:58,974 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#91 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:59,000 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/0551df6be30c48698778ab0572204e85 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0551df6be30c48698778ab0572204e85 2023-04-07 09:53:59,012 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 0551df6be30c48698778ab0572204e85(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:59,012 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:59,012 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=11, startTime=1680861238928; duration=0sec 2023-04-07 09:53:59,012 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:53:59,012 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:53:59,014 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:53:59,014 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:53:59,014 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:53:59,015 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/e98235fd952c4de0a5fabf5d3a375d28, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1adc8b17adb3480aa2fba584413eb4c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ae263c51ab74c1788c2488b70225c77, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/99d1563357af47188101e27610f877f0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/66415c78c7c347c2af2f6b9747f40ac6] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=60.3 K 2023-04-07 09:53:59,015 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e98235fd952c4de0a5fabf5d3a375d28, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=560, earliestPutTs=1680861236379 2023-04-07 09:53:59,016 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1adc8b17adb3480aa2fba584413eb4c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1680861236947 2023-04-07 09:53:59,016 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 5ae263c51ab74c1788c2488b70225c77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680861237095 2023-04-07 09:53:59,017 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 99d1563357af47188101e27610f877f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680861237737 2023-04-07 09:53:59,017 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 66415c78c7c347c2af2f6b9747f40ac6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861238400 2023-04-07 09:53:59,033 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:53:59,049 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/5ab583a37c2740efb66beef5d2b9b56f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ab583a37c2740efb66beef5d2b9b56f 2023-04-07 09:53:59,056 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into 5ab583a37c2740efb66beef5d2b9b56f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:59,056 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:59,056 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=11, startTime=1680861238928; duration=0sec 2023-04-07 09:53:59,056 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:59,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:53:59,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:53:59,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861299163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861299164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861299164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861299164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861299165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861299265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861299271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861299271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861299272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861299272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,402 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/28376ad719a8450bbfbf5f982f90deda as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/28376ad719a8450bbfbf5f982f90deda 2023-04-07 09:53:59,408 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into 28376ad719a8450bbfbf5f982f90deda(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:53:59,408 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:53:59,408 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=11, startTime=1680861238928; duration=0sec 2023-04-07 09:53:59,408 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/3576004b83494d10ad27486169029205 2023-04-07 09:53:59,408 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:53:59,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861299471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861299474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861299474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861299474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861299475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861299775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861299776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861299776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861299777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:53:59,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861299778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:53:59,822 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/22e1716e99b4483b8c95f549c2bab3e3 2023-04-07 09:54:00,234 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/78794d15ef3e417dbef7fd56607b8214 2023-04-07 09:54:00,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/3576004b83494d10ad27486169029205 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/3576004b83494d10ad27486169029205 2023-04-07 09:54:00,246 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/3576004b83494d10ad27486169029205, entries=150, sequenceid=661, filesize=11.9 K 2023-04-07 09:54:00,247 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/22e1716e99b4483b8c95f549c2bab3e3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/22e1716e99b4483b8c95f549c2bab3e3 2023-04-07 09:54:00,253 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/22e1716e99b4483b8c95f549c2bab3e3, entries=150, sequenceid=661, filesize=11.9 K 2023-04-07 09:54:00,255 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/78794d15ef3e417dbef7fd56607b8214 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/78794d15ef3e417dbef7fd56607b8214 2023-04-07 09:54:00,264 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/78794d15ef3e417dbef7fd56607b8214, entries=150, sequenceid=661, filesize=11.9 K 2023-04-07 09:54:00,265 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.15 KB/134300, heapSize ~373.34 KB/382304, currentSize=53.67 KB/54960 for 0b39eaaa997d54d62464302ce4e71f9b in 1309ms, sequenceid=661, compaction requested=false 2023-04-07 09:54:00,265 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:00,265 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:00,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:00,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:00,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:00,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:00,268 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:00,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:00,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:00,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:00,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@677aff85[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:00,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:00,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:00,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:00,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:00,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:00,278 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:00,279 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:00,279 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:00,279 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:00,279 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:00,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:00,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:00,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:00,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:00,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:54:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:00,283 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:00,283 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,283 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:00,283 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:00,283 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:00,283 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:00,284 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:00,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:00,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,292 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:00,292 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:00,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:00,292 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,292 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:00,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:00,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:00,292 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:00,293 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:00,293 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:00,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/5f9996de2724472894f3abc1617c4cca 2023-04-07 09:54:00,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861300296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861300298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861300301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861300303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861300303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/91fdad2fce2a448b86d20cf8366d9df3 2023-04-07 09:54:00,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/c1c74bf7716a4c59a497384a97763e9c 2023-04-07 09:54:00,348 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/5f9996de2724472894f3abc1617c4cca as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5f9996de2724472894f3abc1617c4cca 2023-04-07 09:54:00,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5f9996de2724472894f3abc1617c4cca, entries=150, sequenceid=677, filesize=11.9 K 2023-04-07 09:54:00,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/91fdad2fce2a448b86d20cf8366d9df3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91fdad2fce2a448b86d20cf8366d9df3 2023-04-07 09:54:00,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91fdad2fce2a448b86d20cf8366d9df3, entries=150, sequenceid=677, filesize=11.9 K 2023-04-07 09:54:00,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/c1c74bf7716a4c59a497384a97763e9c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/c1c74bf7716a4c59a497384a97763e9c 2023-04-07 09:54:00,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/c1c74bf7716a4c59a497384a97763e9c, entries=150, sequenceid=677, filesize=11.9 K 2023-04-07 09:54:00,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0b39eaaa997d54d62464302ce4e71f9b in 86ms, sequenceid=677, compaction requested=true 2023-04-07 09:54:00,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:00,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:00,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:00,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:00,369 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:00,369 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:00,371 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:00,371 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:00,371 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:54:00,372 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:54:00,372 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:00,372 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:00,372 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0551df6be30c48698778ab0572204e85, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/22e1716e99b4483b8c95f549c2bab3e3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91fdad2fce2a448b86d20cf8366d9df3] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.7 K 2023-04-07 09:54:00,372 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/28376ad719a8450bbfbf5f982f90deda, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/3576004b83494d10ad27486169029205, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5f9996de2724472894f3abc1617c4cca] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.7 K 2023-04-07 09:54:00,372 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0551df6be30c48698778ab0572204e85, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861238400 2023-04-07 09:54:00,372 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 28376ad719a8450bbfbf5f982f90deda, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861238400 2023-04-07 09:54:00,373 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 22e1716e99b4483b8c95f549c2bab3e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680861238501 2023-04-07 09:54:00,373 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3576004b83494d10ad27486169029205, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680861238501 2023-04-07 09:54:00,373 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 91fdad2fce2a448b86d20cf8366d9df3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680861239163 2023-04-07 09:54:00,373 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5f9996de2724472894f3abc1617c4cca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680861239163 2023-04-07 09:54:00,383 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:00,385 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#100 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:00,393 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:00,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:00,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:00,398 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:00,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:00,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:00,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:00,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:00,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:00,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:00,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:00,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:00,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:00,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:00,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,404 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:00,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:00,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:00,405 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:00,406 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:00,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:00,407 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/7394ad0b84874f79ba2d5a12c62cc096 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7394ad0b84874f79ba2d5a12c62cc096 2023-04-07 09:54:00,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:00,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:00,410 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/2ba0150fcc654f41a99983e0779b0666 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2ba0150fcc654f41a99983e0779b0666 2023-04-07 09:54:00,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861300412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,416 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into 7394ad0b84874f79ba2d5a12c62cc096(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:00,416 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:00,416 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=13, startTime=1680861240369; duration=0sec 2023-04-07 09:54:00,416 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:00,416 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:00,417 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:00,417 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:54:00,417 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 2ba0150fcc654f41a99983e0779b0666(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:00,418 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:00,418 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:00,418 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=13, startTime=1680861240369; duration=0sec 2023-04-07 09:54:00,418 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ab583a37c2740efb66beef5d2b9b56f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/78794d15ef3e417dbef7fd56607b8214, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/c1c74bf7716a4c59a497384a97763e9c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.7 K 2023-04-07 09:54:00,418 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:00,419 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5ab583a37c2740efb66beef5d2b9b56f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861238400 2023-04-07 09:54:00,419 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 78794d15ef3e417dbef7fd56607b8214, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680861238501 2023-04-07 09:54:00,419 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting c1c74bf7716a4c59a497384a97763e9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680861239163 2023-04-07 09:54:00,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861300413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861300414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861300414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861300414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,424 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/e725720158914ae1a9d59956daa2a9c6 2023-04-07 09:54:00,436 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#103 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:00,452 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/7434c034fb83446b80e9c136324b29b7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7434c034fb83446b80e9c136324b29b7 2023-04-07 09:54:00,459 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into 7434c034fb83446b80e9c136324b29b7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:00,459 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:00,459 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=13, startTime=1680861240369; duration=0sec 2023-04-07 09:54:00,460 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:00,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861300515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861300521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861300522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861300522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861300523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861300717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861300723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861300724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861300724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:00,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861300727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,851 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/4dd1b444e43043f7a9fbff6742f48138 2023-04-07 09:54:00,870 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/549cfa8dd8074f6aabdd42314b005cd5 2023-04-07 09:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/e725720158914ae1a9d59956daa2a9c6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e725720158914ae1a9d59956daa2a9c6 2023-04-07 09:54:00,885 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e725720158914ae1a9d59956daa2a9c6, entries=150, sequenceid=696, filesize=11.9 K 2023-04-07 09:54:00,887 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/4dd1b444e43043f7a9fbff6742f48138 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/4dd1b444e43043f7a9fbff6742f48138 2023-04-07 09:54:00,892 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/4dd1b444e43043f7a9fbff6742f48138, entries=150, sequenceid=696, filesize=11.9 K 2023-04-07 09:54:00,903 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/549cfa8dd8074f6aabdd42314b005cd5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/549cfa8dd8074f6aabdd42314b005cd5 2023-04-07 09:54:00,910 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/549cfa8dd8074f6aabdd42314b005cd5, entries=150, sequenceid=696, filesize=11.9 K 2023-04-07 09:54:00,911 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0b39eaaa997d54d62464302ce4e71f9b in 505ms, sequenceid=696, compaction requested=false 2023-04-07 09:54:00,912 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:00,912 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,914 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:00,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63ab99b8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:00,917 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:00,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,924 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:00,924 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,924 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:00,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:00,926 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:00,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:00,931 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:00,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:00,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:00,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:00,931 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:00,933 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,933 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:00,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,933 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:00,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:00,934 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,935 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,939 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:00,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:00,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:00,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:00,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:00,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:00,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:00,940 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:00,940 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:01,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:01,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 09:54:01,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:01,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:01,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:01,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/bad93bd96a094f20b65d9ace2b3fd3d9 2023-04-07 09:54:01,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:01,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861301039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861301040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861301042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861301045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861301045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:01,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:01,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:01,057 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:01,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:01,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:01,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:01,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:01,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:01,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:01,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:01,064 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,064 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:01,064 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:01,064 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:01,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:01,065 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,066 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:01,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1463bb7b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:01,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:01,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,071 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:01,071 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:01,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:01,072 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:01,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:01,075 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:01,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:01,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:01,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:01,075 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:01,081 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,081 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,082 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:01,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:01,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,110 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,110 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:01,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861301145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861301145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861301148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861301149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861301149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,211 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:01,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:01,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:01,215 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:01,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:01,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:01,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,221 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:01,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:01,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:01,224 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,224 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:01,224 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:01,224 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:01,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,226 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:01,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647de164[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:01,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:01,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:01,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,232 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:01,237 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:01,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:01,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:01,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:01,237 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:01,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:01,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,247 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:01,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:01,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,248 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:01,248 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:01,348 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:01,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861301348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861301348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:01,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:01,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:01,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:01,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861301350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,352 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:01,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:01,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:01,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861301351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861301352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,355 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:01,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:01,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:01,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:01,357 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,357 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:01,357 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:01,358 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:01,359 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:01,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54cbacb9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:01,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:01,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:01,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:01,367 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:01,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:01,370 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:01,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:01,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:01,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:01,370 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:01,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,373 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:01,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,376 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:01,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:01,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:01,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:01,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/afa0b3b01620403aa9e5f4f8c2d3ca5b 2023-04-07 09:54:01,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:01,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/776e44fe5cc74bdcad81b64912eb7689 2023-04-07 09:54:01,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:01,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:01,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:01,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:01,481 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:01,481 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:01,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:01,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/bad93bd96a094f20b65d9ace2b3fd3d9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bad93bd96a094f20b65d9ace2b3fd3d9 2023-04-07 09:54:01,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:01,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:01,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:01,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:01,486 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,486 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:01,486 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:01,486 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:01,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bad93bd96a094f20b65d9ace2b3fd3d9, entries=200, sequenceid=713, filesize=14.3 K 2023-04-07 09:54:01,490 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/afa0b3b01620403aa9e5f4f8c2d3ca5b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/afa0b3b01620403aa9e5f4f8c2d3ca5b 2023-04-07 09:54:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:01,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b9289c1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:01,493 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:01,493 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,494 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:01,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:01,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:01,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/afa0b3b01620403aa9e5f4f8c2d3ca5b, entries=150, sequenceid=713, filesize=11.9 K 2023-04-07 09:54:01,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/776e44fe5cc74bdcad81b64912eb7689 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/776e44fe5cc74bdcad81b64912eb7689 2023-04-07 09:54:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:01,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:01,501 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:01,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:01,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:01,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:01,501 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:01,503 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,503 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,503 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/776e44fe5cc74bdcad81b64912eb7689, entries=150, sequenceid=713, filesize=11.9 K 2023-04-07 09:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,504 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 0b39eaaa997d54d62464302ce4e71f9b in 480ms, sequenceid=713, compaction requested=true 2023-04-07 09:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:01,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:01,505 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,505 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:01,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:01,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,506 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,507 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:01,507 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:54:01,507 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,507 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7394ad0b84874f79ba2d5a12c62cc096, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e725720158914ae1a9d59956daa2a9c6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bad93bd96a094f20b65d9ace2b3fd3d9] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=39.2 K 2023-04-07 09:54:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,507 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:01,508 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:54:01,508 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,508 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2ba0150fcc654f41a99983e0779b0666, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/4dd1b444e43043f7a9fbff6742f48138, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/afa0b3b01620403aa9e5f4f8c2d3ca5b] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.8 K 2023-04-07 09:54:01,507 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 7394ad0b84874f79ba2d5a12c62cc096, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680861239163 2023-04-07 09:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,508 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2ba0150fcc654f41a99983e0779b0666, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680861239163 2023-04-07 09:54:01,509 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e725720158914ae1a9d59956daa2a9c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680861240297 2023-04-07 09:54:01,509 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4dd1b444e43043f7a9fbff6742f48138, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680861240297 2023-04-07 09:54:01,509 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting afa0b3b01620403aa9e5f4f8c2d3ca5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680861240412 2023-04-07 09:54:01,510 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bad93bd96a094f20b65d9ace2b3fd3d9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680861240412 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,516 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:01,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:01,516 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,517 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:01,517 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:01,519 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:01,519 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:01,532 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/46889ae1a1914c5eb8368ef020a6131b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/46889ae1a1914c5eb8368ef020a6131b 2023-04-07 09:54:01,534 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/138b832d01cb413db3faaeaf43d9463f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/138b832d01cb413db3faaeaf43d9463f 2023-04-07 09:54:01,537 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 46889ae1a1914c5eb8368ef020a6131b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:01,537 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:01,537 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=13, startTime=1680861241505; duration=0sec 2023-04-07 09:54:01,537 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:01,537 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:01,539 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:01,539 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:54:01,539 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,540 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7434c034fb83446b80e9c136324b29b7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/549cfa8dd8074f6aabdd42314b005cd5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/776e44fe5cc74bdcad81b64912eb7689] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.8 K 2023-04-07 09:54:01,540 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7434c034fb83446b80e9c136324b29b7, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1680861239163 2023-04-07 09:54:01,541 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 549cfa8dd8074f6aabdd42314b005cd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680861240297 2023-04-07 09:54:01,542 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 776e44fe5cc74bdcad81b64912eb7689, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680861240412 2023-04-07 09:54:01,544 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into 138b832d01cb413db3faaeaf43d9463f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:01,544 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:01,544 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=13, startTime=1680861241505; duration=0sec 2023-04-07 09:54:01,544 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:01,549 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#110 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:01,617 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:01,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:01,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:01,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:01,621 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:01,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:01,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:01,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:01,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:01,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:01,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:01,626 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,626 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:01,627 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:54:01,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:01,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:01,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:01,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,639 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/485f392ed5a94143872449dd7ea0f540 2023-04-07 09:54:01,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:01,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:01,694 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/541d32ada9b44d5bb4f3b13ec8d05fa1 2023-04-07 09:54:01,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861301697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861301697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861301699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861301699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861301705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,750 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/7263772058ef4394922b70aef06e0857 2023-04-07 09:54:01,757 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/485f392ed5a94143872449dd7ea0f540 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/485f392ed5a94143872449dd7ea0f540 2023-04-07 09:54:01,765 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/485f392ed5a94143872449dd7ea0f540, entries=150, sequenceid=734, filesize=11.9 K 2023-04-07 09:54:01,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/541d32ada9b44d5bb4f3b13ec8d05fa1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/541d32ada9b44d5bb4f3b13ec8d05fa1 2023-04-07 09:54:01,772 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/541d32ada9b44d5bb4f3b13ec8d05fa1, entries=150, sequenceid=734, filesize=11.9 K 2023-04-07 09:54:01,773 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/7263772058ef4394922b70aef06e0857 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7263772058ef4394922b70aef06e0857 2023-04-07 09:54:01,794 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7263772058ef4394922b70aef06e0857, entries=150, sequenceid=734, filesize=11.9 K 2023-04-07 09:54:01,795 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0b39eaaa997d54d62464302ce4e71f9b in 168ms, sequenceid=734, compaction requested=false 2023-04-07 09:54:01,795 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:01,795 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a34fab[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:01,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:01,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,801 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:01,801 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,801 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:01,804 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:01,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 09:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:01,808 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:01,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:01,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:01,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:01,808 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:01,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:01,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:01,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:01,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:01,810 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,816 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,816 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,817 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:01,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:01,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,822 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:01,822 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:01,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,823 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,823 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861301821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861301823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861301824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861301825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861301825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,923 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:01,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:01,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:01,927 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:01,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:01,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:01,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861301926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861301926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,930 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,930 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861301929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861301930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861301930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:01,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:01,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:01,932 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,932 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:01,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:01,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:01,935 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,935 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,935 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,935 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69def45a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:01,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:01,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,938 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:01,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:01,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,940 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,940 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,940 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:01,943 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:01,943 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:01,943 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:01,943 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:01,943 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:01,945 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,945 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,945 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:01,946 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:01,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:01,954 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:01,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:01,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:01,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:01,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:01,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:01,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:01,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,955 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:01,955 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:01,955 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:01,955 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:01,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:01,975 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/daf1724b9a24480a920c37f588e34633 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/daf1724b9a24480a920c37f588e34633 2023-04-07 09:54:01,980 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into daf1724b9a24480a920c37f588e34633(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:01,981 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:01,981 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=13, startTime=1680861241505; duration=0sec 2023-04-07 09:54:01,981 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:02,055 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:02,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:02,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:02,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:02,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:02,058 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:02,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:02,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:02,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:02,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:02,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:02,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:02,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:02,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:02,063 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,063 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:02,063 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:02,063 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:02,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:02,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:02,063 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:02,064 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,064 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,065 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,065 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f933bc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:02,066 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:02,067 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,068 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:02,068 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,068 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:02,071 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,071 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,071 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,071 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:02,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:02,073 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:02,073 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:02,073 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:02,073 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:02,073 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:02,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,080 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,080 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,084 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:02,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:02,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,085 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:02,085 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:02,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:02,085 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,085 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,085 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,085 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861302131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861302132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861302132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861302133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861302133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,185 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:02,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:02,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:02,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:02,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:02,192 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:02,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:02,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:02,193 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,193 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,194 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,198 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:02,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:02,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:02,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:02,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:02,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:02,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:02,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:02,201 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,201 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:02,202 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:02,202 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,202 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:02,202 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:02,202 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,202 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:02,202 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:02,204 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,204 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,204 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,204 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5a72e3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:02,207 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:02,207 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,208 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:02,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:02,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:02,208 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,208 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:02,210 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:02,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:02,214 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:02,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:02,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:02,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:02,215 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:02,217 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,217 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,217 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,218 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:02,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,231 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:02,231 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:02,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,238 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/f32ba26716a44f0cba0a3cb063011af2 2023-04-07 09:54:02,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/d68989f856d941eda9dd5746b990bf41 2023-04-07 09:54:02,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:02,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:02,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:02,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:02,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:02,335 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:02,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:02,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:02,336 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,336 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,337 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:02,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:02,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:02,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:02,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:02,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:02,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:02,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:02,350 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,350 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:02,350 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:02,350 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:02,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:02,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:02,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:02,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:02,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:02,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67fca8b4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:02,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:02,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,368 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:02,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:02,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:02,375 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:02,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:02,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:02,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:02,375 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:02,377 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,377 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,377 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,378 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,380 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:02,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:02,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,388 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,388 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,388 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,392 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:02,392 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:02,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861302434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861302436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861302437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861302441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861302445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:02,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:02,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:02,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:02,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:02,496 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:02,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:02,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:02,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,502 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:02,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:02,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:02,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:02,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:02,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:02,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,505 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,506 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,506 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:02,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791eec35[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:02,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:02,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,510 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:02,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:02,511 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,512 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:02,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:02,517 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:02,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:02,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:02,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:02,517 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:02,520 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,520 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,531 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:02,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:02,538 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:02,538 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:02,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,638 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:02,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:02,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:02,656 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:02,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:02,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:02,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,658 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,659 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,660 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:02,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:02,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:02,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:02,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:02,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:02,665 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,665 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:02,665 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:02,665 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:02,669 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:02,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50535479[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:02,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:02,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,673 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:02,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:02,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:02,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:02,675 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:02,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:02,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:02,679 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:02,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:02,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:02,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:02,680 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:02,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:02,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,683 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:02,687 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,687 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:02,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:02,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:02,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:02,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:02,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:02,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,688 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:02,688 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:02,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:02,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/3e9c7c663ab54611a06235abf5af5c11 2023-04-07 09:54:02,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/f32ba26716a44f0cba0a3cb063011af2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f32ba26716a44f0cba0a3cb063011af2 2023-04-07 09:54:02,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f32ba26716a44f0cba0a3cb063011af2, entries=250, sequenceid=750, filesize=16.7 K 2023-04-07 09:54:02,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/d68989f856d941eda9dd5746b990bf41 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d68989f856d941eda9dd5746b990bf41 2023-04-07 09:54:02,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d68989f856d941eda9dd5746b990bf41, entries=150, sequenceid=750, filesize=11.9 K 2023-04-07 09:54:02,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/3e9c7c663ab54611a06235abf5af5c11 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/3e9c7c663ab54611a06235abf5af5c11 2023-04-07 09:54:02,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/3e9c7c663ab54611a06235abf5af5c11, entries=150, sequenceid=750, filesize=11.9 K 2023-04-07 09:54:02,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 0b39eaaa997d54d62464302ce4e71f9b in 931ms, sequenceid=750, compaction requested=true 2023-04-07 09:54:02,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:02,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:02,739 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:02,739 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:02,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:02,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:02,741 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:02,741 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:54:02,741 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,741 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/46889ae1a1914c5eb8368ef020a6131b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/541d32ada9b44d5bb4f3b13ec8d05fa1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d68989f856d941eda9dd5746b990bf41] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.9 K 2023-04-07 09:54:02,741 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42664 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:02,741 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:54:02,741 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,741 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/138b832d01cb413db3faaeaf43d9463f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/485f392ed5a94143872449dd7ea0f540, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f32ba26716a44f0cba0a3cb063011af2] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=41.7 K 2023-04-07 09:54:02,742 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 138b832d01cb413db3faaeaf43d9463f, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680861240412 2023-04-07 09:54:02,742 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 46889ae1a1914c5eb8368ef020a6131b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680861240412 2023-04-07 09:54:02,743 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 485f392ed5a94143872449dd7ea0f540, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1680861241038 2023-04-07 09:54:02,743 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 541d32ada9b44d5bb4f3b13ec8d05fa1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1680861241038 2023-04-07 09:54:02,744 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting f32ba26716a44f0cba0a3cb063011af2, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680861241697 2023-04-07 09:54:02,744 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d68989f856d941eda9dd5746b990bf41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680861241697 2023-04-07 09:54:02,753 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#117 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:02,754 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#118 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:02,780 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/3d6c2eb200044017b57d23956cc95928 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3d6c2eb200044017b57d23956cc95928 2023-04-07 09:54:02,786 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/ef2562ea63bd4ca3bc23b7ce4efd7b9f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ef2562ea63bd4ca3bc23b7ce4efd7b9f 2023-04-07 09:54:02,789 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:02,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:02,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:02,796 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into ef2562ea63bd4ca3bc23b7ce4efd7b9f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:02,796 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:02,796 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=13, startTime=1680861242738; duration=0sec 2023-04-07 09:54:02,796 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:02,796 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:02,797 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:02,797 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:54:02,797 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,797 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/daf1724b9a24480a920c37f588e34633, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7263772058ef4394922b70aef06e0857, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/3e9c7c663ab54611a06235abf5af5c11] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=36.9 K 2023-04-07 09:54:02,798 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting daf1724b9a24480a920c37f588e34633, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680861240412 2023-04-07 09:54:02,798 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 7263772058ef4394922b70aef06e0857, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1680861241038 2023-04-07 09:54:02,799 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3e9c7c663ab54611a06235abf5af5c11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680861241697 2023-04-07 09:54:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:02,800 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:02,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:02,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:02,801 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,801 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,803 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:02,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:02,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:02,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:02,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:02,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:02,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:02,805 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:02,805 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:02,805 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-07 09:54:02,806 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:02,807 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:02,807 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:02,807 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:02,807 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:02,807 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:02,809 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 3d6c2eb200044017b57d23956cc95928(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:02,809 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:02,809 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=13, startTime=1680861242739; duration=0sec 2023-04-07 09:54:02,810 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:02,825 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#119 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:02,863 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/00f7388a61c744678a9c03080375585e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/00f7388a61c744678a9c03080375585e 2023-04-07 09:54:02,869 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into 00f7388a61c744678a9c03080375585e(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:02,869 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:02,869 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=13, startTime=1680861242739; duration=0sec 2023-04-07 09:54:02,870 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:02,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:02,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:02,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861302949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861302949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861302950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861302950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:02,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:02,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861302952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861303053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861303053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861303053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861303056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861303256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861303260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,265 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/2478e209627b43aaa9589f4c37d18774 2023-04-07 09:54:03,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861303264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861303264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861303558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861303564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861303570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861303570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:03,703 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/c89d738c06b640e8bad8e1e7151ac716 2023-04-07 09:54:03,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:03,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57366 deadline: 1680861303955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:04,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861304063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:04,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861304072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:04,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:04,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861304076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861304076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,116 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/5c720610e5a54ec494fb0568b93775af 2023-04-07 09:54:04,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/2478e209627b43aaa9589f4c37d18774 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2478e209627b43aaa9589f4c37d18774 2023-04-07 09:54:04,126 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2478e209627b43aaa9589f4c37d18774, entries=150, sequenceid=769, filesize=11.9 K 2023-04-07 09:54:04,127 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/c89d738c06b640e8bad8e1e7151ac716 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c89d738c06b640e8bad8e1e7151ac716 2023-04-07 09:54:04,132 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c89d738c06b640e8bad8e1e7151ac716, entries=150, sequenceid=769, filesize=11.9 K 2023-04-07 09:54:04,133 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/5c720610e5a54ec494fb0568b93775af as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5c720610e5a54ec494fb0568b93775af 2023-04-07 09:54:04,143 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5c720610e5a54ec494fb0568b93775af, entries=150, sequenceid=769, filesize=11.9 K 2023-04-07 09:54:04,145 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 0b39eaaa997d54d62464302ce4e71f9b in 1340ms, sequenceid=769, compaction requested=false 2023-04-07 09:54:04,145 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:04,145 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:04,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:04,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:04,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:04,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:04,149 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,149 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cd2f9fa[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:04,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:04,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,153 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:04,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:04,154 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,154 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:04,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:04,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:04,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:04,159 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:04,159 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:04,159 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:04,159 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:04,159 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:04,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:04,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:04,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:04,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:04,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:04,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:04,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:04,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:04,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:04,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:04,174 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:04,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:04,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:04,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:04,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:04,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:04,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:04,277 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:04,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:04,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:04,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:04,278 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,278 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:04,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:04,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:04,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:04,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:04,284 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:04,284 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:04,284 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:04,284 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-07 09:54:04,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:04,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:04,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:04,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:04,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:04,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:04,301 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/1c90d7bd2d784ac8a9fc50fa552bc08a 2023-04-07 09:54:04,313 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/0ebc0bed7b254759a3bf44635d12ca04 2023-04-07 09:54:04,730 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/d4457710e24f4c679ce20e3e09156cf4 2023-04-07 09:54:04,735 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/1c90d7bd2d784ac8a9fc50fa552bc08a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/1c90d7bd2d784ac8a9fc50fa552bc08a 2023-04-07 09:54:04,739 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/1c90d7bd2d784ac8a9fc50fa552bc08a, entries=150, sequenceid=786, filesize=11.9 K 2023-04-07 09:54:04,740 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/0ebc0bed7b254759a3bf44635d12ca04 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0ebc0bed7b254759a3bf44635d12ca04 2023-04-07 09:54:04,749 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0ebc0bed7b254759a3bf44635d12ca04, entries=150, sequenceid=786, filesize=11.9 K 2023-04-07 09:54:04,753 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/d4457710e24f4c679ce20e3e09156cf4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/d4457710e24f4c679ce20e3e09156cf4 2023-04-07 09:54:04,758 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/d4457710e24f4c679ce20e3e09156cf4, entries=150, sequenceid=786, filesize=11.9 K 2023-04-07 09:54:04,759 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 0b39eaaa997d54d62464302ce4e71f9b in 475ms, sequenceid=786, compaction requested=true 2023-04-07 09:54:04,759 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:04,759 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:04,761 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,761 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:04,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e694bdc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:04,763 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:04,763 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,764 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,765 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:04,765 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:04,765 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:04,765 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,765 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:04,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:04,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:04,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:04,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:04,770 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:04,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:04,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:04,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:04,770 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:04,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,772 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:04,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:04,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:04,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:04,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:04,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,777 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,777 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,777 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,777 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:04,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:04,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:04,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:04,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:04,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:04,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:04,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:04,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:04,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:04,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:04,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:04,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:04,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:04,882 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:04,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:04,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:04,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:04,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:04,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:04,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:04,884 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:04,884 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:04,885 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:04,885 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:04,887 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:04,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:04,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10987360[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:04,889 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:04,889 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,890 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:04,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:04,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:04,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:04,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:04,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:04,895 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:04,895 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:04,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:04,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:04,896 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:04,897 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,897 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:04,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,897 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:04,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,900 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:04,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:04,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:04,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:04,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:04,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:04,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:04,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:04,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:04,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:04,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:04,904 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:04,904 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:05,005 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:05,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:05,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:05,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:05,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:05,008 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:05,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:05,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:05,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,010 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:05,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:05,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:05,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:05,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:05,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:05,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:05,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:05,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:05,015 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,016 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,016 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:05,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aa20d96[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:05,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:05,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:05,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:05,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:05,019 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,019 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:05,020 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:05,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,021 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:05,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:05,023 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:05,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:05,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:05,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:05,023 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:05,024 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,024 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,024 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:05,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,029 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:05,029 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:05,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:05,084 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:54:05,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:05,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:05,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:05,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:05,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:05,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:05,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=798 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/bc1d898037fb4bc482a9752ce54d946b 2023-04-07 09:54:05,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680861245088, max timestamp=1680861245109Num uniques 859; 2023-04-07 09:54:05,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-07 09:54:05,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680861245088, max timestamp=1680861245109Num uniques 859; 2023-04-07 09:54:05,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-07 09:54:05,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680861245088, max timestamp=1680861245109Num uniques 859; 2023-04-07 09:54:05,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-07 09:54:05,130 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:05,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:05,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:05,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861305128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861305130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:05,133 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:05,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,133 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861305130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,134 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:05,134 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:05,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861305131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,134 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,134 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:05,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:05,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:05,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:05,138 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:05,139 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,139 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:05,140 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,140 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:05,140 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:05,140 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:05,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:05,141 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,141 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,142 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,142 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:05,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51343211[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:05,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:05,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,149 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:05,149 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,149 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:05,149 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:05,150 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:05,152 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:05,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:05,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:05,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:05,153 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:05,154 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,154 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,154 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,155 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,156 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,158 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:05,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:05,159 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:05,159 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:05,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861305233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861305234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861305240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861305244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,259 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:05,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:05,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:05,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:05,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:05,264 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:05,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:05,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:05,265 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,265 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,267 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,267 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:05,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:05,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:05,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:05,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:05,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:05,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:05,269 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,269 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:05,270 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:05,270 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:05,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:05,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:05,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:05,271 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:05,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557b440[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:05,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:05,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,275 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:05,275 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:05,275 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,276 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:05,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:05,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:05,277 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:05,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:05,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:05,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:05,277 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:05,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,281 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:05,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,283 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,283 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:05,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:05,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,384 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:05,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:05,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:05,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:05,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:05,387 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:05,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:05,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:05,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,389 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,389 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,389 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,389 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:05,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:05,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:05,391 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,391 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:05,391 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:05,391 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:05,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:05,393 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:05,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5607d4f9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:05,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:05,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,397 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,397 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,397 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:05,397 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:05,398 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:05,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,398 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,398 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:05,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:05,401 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:05,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:05,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:05,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:05,401 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:05,402 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,402 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,402 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,404 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:05,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:05,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:05,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:05,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:05,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57306 deadline: 1680861305441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57350 deadline: 1680861305442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57336 deadline: 1680861305442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:05,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57324 deadline: 1680861305446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,507 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:05,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:05,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:05,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:05,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:05,510 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:05,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:05,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:05,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,513 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:05,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:05,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:05,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:05,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:05,515 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,515 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. started... 2023-04-07 09:54:05,515 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. as already flushing 2023-04-07 09:54:05,515 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:05,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:05,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=798 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/35fe938aa7904d3bacdeac19e5e627d3 2023-04-07 09:54:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:05,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c83bd12[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:05,519 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:05,519 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:05,537 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,537 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:05,543 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=798 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/089f0a26e4e04860b1119999ed1aec59 2023-04-07 09:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:05,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:05,547 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:05,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:05,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:05,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:05,547 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:05,550 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,550 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/bc1d898037fb4bc482a9752ce54d946b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bc1d898037fb4bc482a9752ce54d946b 2023-04-07 09:54:05,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:05,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bc1d898037fb4bc482a9752ce54d946b, entries=250, sequenceid=798, filesize=16.7 K 2023-04-07 09:54:05,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/35fe938aa7904d3bacdeac19e5e627d3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/35fe938aa7904d3bacdeac19e5e627d3 2023-04-07 09:54:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:05,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,564 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/35fe938aa7904d3bacdeac19e5e627d3, entries=150, sequenceid=798, filesize=11.9 K 2023-04-07 09:54:05,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/089f0a26e4e04860b1119999ed1aec59 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/089f0a26e4e04860b1119999ed1aec59 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:05,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:05,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:05,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:05,569 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:05,569 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:05,569 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/089f0a26e4e04860b1119999ed1aec59, entries=150, sequenceid=798, filesize=11.9 K 2023-04-07 09:54:05,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:05,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 0b39eaaa997d54d62464302ce4e71f9b in 486ms, sequenceid=798, compaction requested=true 2023-04-07 09:54:05,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:05,570 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:05,570 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:05,570 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:05,570 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:05,570 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:05,572 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54952 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:05,572 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/A is initiating minor compaction (all files) 2023-04-07 09:54:05,572 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50072 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:05,572 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/A in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,572 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/B is initiating minor compaction (all files) 2023-04-07 09:54:05,572 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ef2562ea63bd4ca3bc23b7ce4efd7b9f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2478e209627b43aaa9589f4c37d18774, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/1c90d7bd2d784ac8a9fc50fa552bc08a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bc1d898037fb4bc482a9752ce54d946b] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=53.7 K 2023-04-07 09:54:05,572 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/B in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,572 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3d6c2eb200044017b57d23956cc95928, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c89d738c06b640e8bad8e1e7151ac716, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0ebc0bed7b254759a3bf44635d12ca04, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/35fe938aa7904d3bacdeac19e5e627d3] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=48.9 K 2023-04-07 09:54:05,573 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ef2562ea63bd4ca3bc23b7ce4efd7b9f, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680861241697 2023-04-07 09:54:05,573 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2478e209627b43aaa9589f4c37d18774, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=769, earliestPutTs=1680861241819 2023-04-07 09:54:05,573 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3d6c2eb200044017b57d23956cc95928, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680861241697 2023-04-07 09:54:05,574 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1c90d7bd2d784ac8a9fc50fa552bc08a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680861242945 2023-04-07 09:54:05,574 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting c89d738c06b640e8bad8e1e7151ac716, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=769, earliestPutTs=1680861241819 2023-04-07 09:54:05,574 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting bc1d898037fb4bc482a9752ce54d946b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=798, earliestPutTs=1680861245072 2023-04-07 09:54:05,575 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 0ebc0bed7b254759a3bf44635d12ca04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680861242945 2023-04-07 09:54:05,575 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 35fe938aa7904d3bacdeac19e5e627d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=798, earliestPutTs=1680861245078 2023-04-07 09:54:05,586 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#A#compaction#129 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:05,587 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#B#compaction#130 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:05,593 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:49660 2023-04-07 09:54:05,593 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,594 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:49660 2023-04-07 09:54:05,594 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,598 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:49660 2023-04-07 09:54:05,598 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,602 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:49660 2023-04-07 09:54:05,602 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,603 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/559f9bfd61184449a414c1092e817ce9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/559f9bfd61184449a414c1092e817ce9 2023-04-07 09:54:05,608 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/B of 0b39eaaa997d54d62464302ce4e71f9b into 559f9bfd61184449a414c1092e817ce9(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:05,608 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:05,608 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/B, priority=12, startTime=1680861245570; duration=0sec 2023-04-07 09:54:05,608 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:05,608 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:05,609 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50072 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:05,610 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 0b39eaaa997d54d62464302ce4e71f9b/C is initiating minor compaction (all files) 2023-04-07 09:54:05,610 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0b39eaaa997d54d62464302ce4e71f9b/C in TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:05,610 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/00f7388a61c744678a9c03080375585e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5c720610e5a54ec494fb0568b93775af, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/d4457710e24f4c679ce20e3e09156cf4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/089f0a26e4e04860b1119999ed1aec59] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp, totalSize=48.9 K 2023-04-07 09:54:05,610 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 00f7388a61c744678a9c03080375585e, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680861241697 2023-04-07 09:54:05,611 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5c720610e5a54ec494fb0568b93775af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=769, earliestPutTs=1680861241819 2023-04-07 09:54:05,611 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d4457710e24f4c679ce20e3e09156cf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680861242945 2023-04-07 09:54:05,611 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 089f0a26e4e04860b1119999ed1aec59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=798, earliestPutTs=1680861245078 2023-04-07 09:54:05,617 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0b39eaaa997d54d62464302ce4e71f9b#C#compaction#131 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:05,669 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:05,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:05,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:05,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=161.02 KB heapSize=445.92 KB 2023-04-07 09:54:05,750 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:49660 2023-04-07 09:54:05,751 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:49660 2023-04-07 09:54:05,751 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:05,751 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:05,751 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:49660 2023-04-07 09:54:05,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:05,751 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:05,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:05,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:05,752 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:49660 2023-04-07 09:54:05,752 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,968 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:49660 2023-04-07 09:54:05,968 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 213 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 09:54:05,968 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4241 2023-04-07 09:54:05,969 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4172 2023-04-07 09:54:05,969 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 09:54:05,969 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 1861 2023-04-07 09:54:05,969 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 5579 rows 2023-04-07 09:54:05,969 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 1858 2023-04-07 09:54:05,969 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 5573 rows 2023-04-07 09:54:05,969 INFO [Listener at localhost/39511] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 09:54:05,969 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:49660 2023-04-07 09:54:05,969 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:05,972 INFO [Listener at localhost/39511] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 09:54:05,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 09:54:05,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:05,990 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861245989"}]},"ts":"1680861245989"} 2023-04-07 09:54:05,991 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 09:54:05,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 09:54:05,993 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 09:54:05,995 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b39eaaa997d54d62464302ce4e71f9b, UNASSIGN}] 2023-04-07 09:54:05,997 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b39eaaa997d54d62464302ce4e71f9b, UNASSIGN 2023-04-07 09:54:05,997 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=0b39eaaa997d54d62464302ce4e71f9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:05,998 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861245997"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861245997"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861245997"}]},"ts":"1680861245997"} 2023-04-07 09:54:06,000 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:54:06,004 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/f4960f657e4f4fa5bf7b4b3a5215d96a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f4960f657e4f4fa5bf7b4b3a5215d96a 2023-04-07 09:54:06,009 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/A of 0b39eaaa997d54d62464302ce4e71f9b into f4960f657e4f4fa5bf7b4b3a5215d96a(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:06,009 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:06,009 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/A, priority=12, startTime=1680861245570; duration=0sec 2023-04-07 09:54:06,010 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:06,026 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/354c4e682cb649eda4184491403ddd12 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/354c4e682cb649eda4184491403ddd12 2023-04-07 09:54:06,030 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0b39eaaa997d54d62464302ce4e71f9b/C of 0b39eaaa997d54d62464302ce4e71f9b into 354c4e682cb649eda4184491403ddd12(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:06,031 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:06,031 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b., storeName=0b39eaaa997d54d62464302ce4e71f9b/C, priority=12, startTime=1680861245570; duration=0sec 2023-04-07 09:54:06,031 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:06,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 09:54:06,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:06,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=836 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/2ea4eab5fcdb41a99ea4df918a1b5e23 2023-04-07 09:54:06,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0b39eaaa997d54d62464302ce4e71f9b, disabling compactions & flushes 2023-04-07 09:54:06,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:06,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 09:54:06,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=836 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/7bc48506bae84d2bbb1c1e6052fa77ce 2023-04-07 09:54:06,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 09:54:06,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=836 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/466b0d721aed432aa59cd68173cd7795 2023-04-07 09:54:06,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/2ea4eab5fcdb41a99ea4df918a1b5e23 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2ea4eab5fcdb41a99ea4df918a1b5e23 2023-04-07 09:54:06,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2ea4eab5fcdb41a99ea4df918a1b5e23, entries=150, sequenceid=836, filesize=11.9 K 2023-04-07 09:54:06,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/7bc48506bae84d2bbb1c1e6052fa77ce as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7bc48506bae84d2bbb1c1e6052fa77ce 2023-04-07 09:54:06,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7bc48506bae84d2bbb1c1e6052fa77ce, entries=150, sequenceid=836, filesize=11.9 K 2023-04-07 09:54:06,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/466b0d721aed432aa59cd68173cd7795 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/466b0d721aed432aa59cd68173cd7795 2023-04-07 09:54:07,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/466b0d721aed432aa59cd68173cd7795, entries=150, sequenceid=836, filesize=11.9 K 2023-04-07 09:54:07,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.88 KB/456576, currentSize=13.42 KB/13740 for 0b39eaaa997d54d62464302ce4e71f9b in 1253ms, sequenceid=836, compaction requested=false 2023-04-07 09:54:07,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:07,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. after waiting 0 ms 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:07,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0b39eaaa997d54d62464302ce4e71f9b 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=A 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=B 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0b39eaaa997d54d62464302ce4e71f9b, store=C 2023-04-07 09:54:07,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:07,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 09:54:07,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/98ef0fc99c374580ac2c623786ddd69a 2023-04-07 09:54:07,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/429172c23d984d3f82525e2a6521a6db 2023-04-07 09:54:08,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 09:54:08,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/365d31fab67b44058f2e9a4b0be579dc 2023-04-07 09:54:08,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/A/98ef0fc99c374580ac2c623786ddd69a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/98ef0fc99c374580ac2c623786ddd69a 2023-04-07 09:54:08,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/98ef0fc99c374580ac2c623786ddd69a, entries=100, sequenceid=843, filesize=9.5 K 2023-04-07 09:54:08,242 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/B/429172c23d984d3f82525e2a6521a6db as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/429172c23d984d3f82525e2a6521a6db 2023-04-07 09:54:08,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/429172c23d984d3f82525e2a6521a6db, entries=100, sequenceid=843, filesize=9.5 K 2023-04-07 09:54:08,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/.tmp/C/365d31fab67b44058f2e9a4b0be579dc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/365d31fab67b44058f2e9a4b0be579dc 2023-04-07 09:54:08,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/365d31fab67b44058f2e9a4b0be579dc, entries=100, sequenceid=843, filesize=9.5 K 2023-04-07 09:54:08,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 0b39eaaa997d54d62464302ce4e71f9b in 1247ms, sequenceid=843, compaction requested=true 2023-04-07 09:54:08,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7b5513d5d7194a19b887a8ee9c03f728, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/66c09c8b213e45458bf903883a024d2b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/779c957179474c60aaaec851f39f2e88, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/8e3c8337d8714fd4b872c5cdf24f2e7f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/51b095144ab4406ebecaab01c88c7cb3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/fcf8cb982f2d4469b48619324a12c4ca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/565415bf02f949dc814d506b518709db, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/c5d30d1c02184cfea0a69a486ba83bb2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7991f9fb5edf421a97dde03de5b5afc0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/08f611ac931b4ced825d3d0d8f7ec144, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/d702393a707a40b895bb2bc47403d9ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5284a66e109d46f291319eb10099b58d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ea9df95c335a4b52b2f95bf970d85e0d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/96f4952e59c84c198800c769fc440825, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/eef9f380433d4ea0ba3807fdced6c514, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/65e9686807d74837bb95118daa3b8c91, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/06bf4464fa2748bf81b8dc455b2b295a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/525916fa01aa4f90bf0bddb35b399b1f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/76872dbd8e00435a9223a7b3f5d1ac06, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/683a1d55c1144f4c87acc703d4059479, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d57f4b4fc142c5b9f276ebea520d4c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5ad1b6c28ade4cb5bfc775c61c9ec789, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/b3e088279bba437a9dfa6789958e2d4d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/99416ef10aaa42cb9c4b857cba2258f9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e3103480eb2f4eebb9ab21cdb6a590ba, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f102afb9d9664a59b30b2e360b5375f2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e42f1d25289149b8b3f7d8ebca1fdbd0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/82b357cf70e04db485f23a103977d1fa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/52b9f49bdb224f8289887b8213db8c29, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d1969904de4784b5257df5959ef291, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/28376ad719a8450bbfbf5f982f90deda, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/3576004b83494d10ad27486169029205, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7394ad0b84874f79ba2d5a12c62cc096, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5f9996de2724472894f3abc1617c4cca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e725720158914ae1a9d59956daa2a9c6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bad93bd96a094f20b65d9ace2b3fd3d9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/138b832d01cb413db3faaeaf43d9463f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/485f392ed5a94143872449dd7ea0f540, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f32ba26716a44f0cba0a3cb063011af2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ef2562ea63bd4ca3bc23b7ce4efd7b9f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2478e209627b43aaa9589f4c37d18774, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/1c90d7bd2d784ac8a9fc50fa552bc08a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bc1d898037fb4bc482a9752ce54d946b] to archive 2023-04-07 09:54:08,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:54:08,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7b5513d5d7194a19b887a8ee9c03f728 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7b5513d5d7194a19b887a8ee9c03f728 2023-04-07 09:54:08,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/66c09c8b213e45458bf903883a024d2b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/66c09c8b213e45458bf903883a024d2b 2023-04-07 09:54:08,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/779c957179474c60aaaec851f39f2e88 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/779c957179474c60aaaec851f39f2e88 2023-04-07 09:54:08,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/8e3c8337d8714fd4b872c5cdf24f2e7f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/8e3c8337d8714fd4b872c5cdf24f2e7f 2023-04-07 09:54:08,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/51b095144ab4406ebecaab01c88c7cb3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/51b095144ab4406ebecaab01c88c7cb3 2023-04-07 09:54:08,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/fcf8cb982f2d4469b48619324a12c4ca to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/fcf8cb982f2d4469b48619324a12c4ca 2023-04-07 09:54:08,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/565415bf02f949dc814d506b518709db to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/565415bf02f949dc814d506b518709db 2023-04-07 09:54:08,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/c5d30d1c02184cfea0a69a486ba83bb2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/c5d30d1c02184cfea0a69a486ba83bb2 2023-04-07 09:54:08,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7991f9fb5edf421a97dde03de5b5afc0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7991f9fb5edf421a97dde03de5b5afc0 2023-04-07 09:54:08,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/08f611ac931b4ced825d3d0d8f7ec144 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/08f611ac931b4ced825d3d0d8f7ec144 2023-04-07 09:54:08,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/d702393a707a40b895bb2bc47403d9ed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/d702393a707a40b895bb2bc47403d9ed 2023-04-07 09:54:08,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5284a66e109d46f291319eb10099b58d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5284a66e109d46f291319eb10099b58d 2023-04-07 09:54:08,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ea9df95c335a4b52b2f95bf970d85e0d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ea9df95c335a4b52b2f95bf970d85e0d 2023-04-07 09:54:08,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/96f4952e59c84c198800c769fc440825 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/96f4952e59c84c198800c769fc440825 2023-04-07 09:54:08,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/eef9f380433d4ea0ba3807fdced6c514 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/eef9f380433d4ea0ba3807fdced6c514 2023-04-07 09:54:08,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/65e9686807d74837bb95118daa3b8c91 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/65e9686807d74837bb95118daa3b8c91 2023-04-07 09:54:08,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/06bf4464fa2748bf81b8dc455b2b295a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/06bf4464fa2748bf81b8dc455b2b295a 2023-04-07 09:54:08,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/525916fa01aa4f90bf0bddb35b399b1f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/525916fa01aa4f90bf0bddb35b399b1f 2023-04-07 09:54:08,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/76872dbd8e00435a9223a7b3f5d1ac06 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/76872dbd8e00435a9223a7b3f5d1ac06 2023-04-07 09:54:08,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/683a1d55c1144f4c87acc703d4059479 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/683a1d55c1144f4c87acc703d4059479 2023-04-07 09:54:08,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d57f4b4fc142c5b9f276ebea520d4c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d57f4b4fc142c5b9f276ebea520d4c 2023-04-07 09:54:08,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5ad1b6c28ade4cb5bfc775c61c9ec789 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5ad1b6c28ade4cb5bfc775c61c9ec789 2023-04-07 09:54:08,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/b3e088279bba437a9dfa6789958e2d4d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/b3e088279bba437a9dfa6789958e2d4d 2023-04-07 09:54:08,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/99416ef10aaa42cb9c4b857cba2258f9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/99416ef10aaa42cb9c4b857cba2258f9 2023-04-07 09:54:08,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e3103480eb2f4eebb9ab21cdb6a590ba to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e3103480eb2f4eebb9ab21cdb6a590ba 2023-04-07 09:54:08,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f102afb9d9664a59b30b2e360b5375f2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f102afb9d9664a59b30b2e360b5375f2 2023-04-07 09:54:08,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e42f1d25289149b8b3f7d8ebca1fdbd0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e42f1d25289149b8b3f7d8ebca1fdbd0 2023-04-07 09:54:08,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/82b357cf70e04db485f23a103977d1fa to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/82b357cf70e04db485f23a103977d1fa 2023-04-07 09:54:08,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/52b9f49bdb224f8289887b8213db8c29 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/52b9f49bdb224f8289887b8213db8c29 2023-04-07 09:54:08,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d1969904de4784b5257df5959ef291 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/63d1969904de4784b5257df5959ef291 2023-04-07 09:54:08,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/28376ad719a8450bbfbf5f982f90deda to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/28376ad719a8450bbfbf5f982f90deda 2023-04-07 09:54:08,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/3576004b83494d10ad27486169029205 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/3576004b83494d10ad27486169029205 2023-04-07 09:54:08,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7394ad0b84874f79ba2d5a12c62cc096 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/7394ad0b84874f79ba2d5a12c62cc096 2023-04-07 09:54:08,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5f9996de2724472894f3abc1617c4cca to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/5f9996de2724472894f3abc1617c4cca 2023-04-07 09:54:08,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e725720158914ae1a9d59956daa2a9c6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/e725720158914ae1a9d59956daa2a9c6 2023-04-07 09:54:08,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bad93bd96a094f20b65d9ace2b3fd3d9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bad93bd96a094f20b65d9ace2b3fd3d9 2023-04-07 09:54:08,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/138b832d01cb413db3faaeaf43d9463f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/138b832d01cb413db3faaeaf43d9463f 2023-04-07 09:54:08,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/485f392ed5a94143872449dd7ea0f540 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/485f392ed5a94143872449dd7ea0f540 2023-04-07 09:54:08,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f32ba26716a44f0cba0a3cb063011af2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f32ba26716a44f0cba0a3cb063011af2 2023-04-07 09:54:08,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ef2562ea63bd4ca3bc23b7ce4efd7b9f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/ef2562ea63bd4ca3bc23b7ce4efd7b9f 2023-04-07 09:54:08,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2478e209627b43aaa9589f4c37d18774 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2478e209627b43aaa9589f4c37d18774 2023-04-07 09:54:08,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/1c90d7bd2d784ac8a9fc50fa552bc08a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/1c90d7bd2d784ac8a9fc50fa552bc08a 2023-04-07 09:54:08,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bc1d898037fb4bc482a9752ce54d946b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/bc1d898037fb4bc482a9752ce54d946b 2023-04-07 09:54:08,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a02cf64048da4621b5dc17d7de4aa1cb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7cea7b6ac6354e95b3cc0e722096d75e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/87044303560b45518e7cb5c7a98cfd10, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/14acc2cff502435bbeba80019405b5d0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/117a7728c3334c87a2247a91fc3a0fb3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d1ed7bfa2a444fe5b65bfaf72d23750d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/b7120c8b81694b1199ebbac71a1c1cd7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/70ae3778a5e74344a339567e37491966, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c25a79ce98c544c699d6a91dbe41ef5e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/fb8e65500b7148a5ba62bf0555aa4313, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0896455ab8f842e99efa8ac5af7d3dc2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/9017511772364f8198865310b3f2e066, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1216b87b20ee4c21b34f2f92c94255e4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/23b8a1d806e949dc8d5c0a6983b12d54, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91bf2699f6204fc0b8da42c376cfe320, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/880addf3e2184be0869a89278efa530c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/ff8a562ab2574302b2202ab4fe5fe85c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/cd11eff0026a4932a639452f241675bb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/bd74e6bd25f544eb8ff1243b520aeffe, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c15c12f9ef6a43718c97c1ac3ea0c950, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1fd3ea69a21e49bd8c4cb27de38773fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a3915e9e512b4a50994bcfad667b03c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/f30038c3ee274591a37ac12ae91a7c6c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7e7b912f8fc0470bbd4a68acc2a546df, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/5b282fdb80b54968adc4aa9d071d039d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/aaf3a0253803449bbe8b8af7cfe07285, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/6ec1134433b2409f869be3466222f6f4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a101ae942b2d4d398b97ef9ff1038375, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2d60e6b0e3a64cddaad922358e1e27da, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0551df6be30c48698778ab0572204e85, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3218114cbce54659ae3c5f30eb6d368a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/22e1716e99b4483b8c95f549c2bab3e3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2ba0150fcc654f41a99983e0779b0666, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91fdad2fce2a448b86d20cf8366d9df3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/4dd1b444e43043f7a9fbff6742f48138, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/46889ae1a1914c5eb8368ef020a6131b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/afa0b3b01620403aa9e5f4f8c2d3ca5b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/541d32ada9b44d5bb4f3b13ec8d05fa1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3d6c2eb200044017b57d23956cc95928, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d68989f856d941eda9dd5746b990bf41, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c89d738c06b640e8bad8e1e7151ac716, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0ebc0bed7b254759a3bf44635d12ca04, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/35fe938aa7904d3bacdeac19e5e627d3] to archive 2023-04-07 09:54:08,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:54:08,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a02cf64048da4621b5dc17d7de4aa1cb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a02cf64048da4621b5dc17d7de4aa1cb 2023-04-07 09:54:08,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7cea7b6ac6354e95b3cc0e722096d75e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7cea7b6ac6354e95b3cc0e722096d75e 2023-04-07 09:54:08,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/87044303560b45518e7cb5c7a98cfd10 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/87044303560b45518e7cb5c7a98cfd10 2023-04-07 09:54:08,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/14acc2cff502435bbeba80019405b5d0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/14acc2cff502435bbeba80019405b5d0 2023-04-07 09:54:08,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/117a7728c3334c87a2247a91fc3a0fb3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/117a7728c3334c87a2247a91fc3a0fb3 2023-04-07 09:54:08,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d1ed7bfa2a444fe5b65bfaf72d23750d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d1ed7bfa2a444fe5b65bfaf72d23750d 2023-04-07 09:54:08,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/b7120c8b81694b1199ebbac71a1c1cd7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/b7120c8b81694b1199ebbac71a1c1cd7 2023-04-07 09:54:08,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/70ae3778a5e74344a339567e37491966 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/70ae3778a5e74344a339567e37491966 2023-04-07 09:54:08,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c25a79ce98c544c699d6a91dbe41ef5e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c25a79ce98c544c699d6a91dbe41ef5e 2023-04-07 09:54:08,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/fb8e65500b7148a5ba62bf0555aa4313 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/fb8e65500b7148a5ba62bf0555aa4313 2023-04-07 09:54:08,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0896455ab8f842e99efa8ac5af7d3dc2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0896455ab8f842e99efa8ac5af7d3dc2 2023-04-07 09:54:08,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/9017511772364f8198865310b3f2e066 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/9017511772364f8198865310b3f2e066 2023-04-07 09:54:08,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1216b87b20ee4c21b34f2f92c94255e4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1216b87b20ee4c21b34f2f92c94255e4 2023-04-07 09:54:08,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/23b8a1d806e949dc8d5c0a6983b12d54 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/23b8a1d806e949dc8d5c0a6983b12d54 2023-04-07 09:54:08,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91bf2699f6204fc0b8da42c376cfe320 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91bf2699f6204fc0b8da42c376cfe320 2023-04-07 09:54:08,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/880addf3e2184be0869a89278efa530c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/880addf3e2184be0869a89278efa530c 2023-04-07 09:54:08,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/ff8a562ab2574302b2202ab4fe5fe85c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/ff8a562ab2574302b2202ab4fe5fe85c 2023-04-07 09:54:08,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/cd11eff0026a4932a639452f241675bb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/cd11eff0026a4932a639452f241675bb 2023-04-07 09:54:08,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/bd74e6bd25f544eb8ff1243b520aeffe to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/bd74e6bd25f544eb8ff1243b520aeffe 2023-04-07 09:54:08,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c15c12f9ef6a43718c97c1ac3ea0c950 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c15c12f9ef6a43718c97c1ac3ea0c950 2023-04-07 09:54:08,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1fd3ea69a21e49bd8c4cb27de38773fd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/1fd3ea69a21e49bd8c4cb27de38773fd 2023-04-07 09:54:08,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a3915e9e512b4a50994bcfad667b03c1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a3915e9e512b4a50994bcfad667b03c1 2023-04-07 09:54:08,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/f30038c3ee274591a37ac12ae91a7c6c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/f30038c3ee274591a37ac12ae91a7c6c 2023-04-07 09:54:08,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7e7b912f8fc0470bbd4a68acc2a546df to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7e7b912f8fc0470bbd4a68acc2a546df 2023-04-07 09:54:08,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/5b282fdb80b54968adc4aa9d071d039d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/5b282fdb80b54968adc4aa9d071d039d 2023-04-07 09:54:08,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/aaf3a0253803449bbe8b8af7cfe07285 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/aaf3a0253803449bbe8b8af7cfe07285 2023-04-07 09:54:08,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/6ec1134433b2409f869be3466222f6f4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/6ec1134433b2409f869be3466222f6f4 2023-04-07 09:54:08,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a101ae942b2d4d398b97ef9ff1038375 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/a101ae942b2d4d398b97ef9ff1038375 2023-04-07 09:54:08,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2d60e6b0e3a64cddaad922358e1e27da to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2d60e6b0e3a64cddaad922358e1e27da 2023-04-07 09:54:08,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0551df6be30c48698778ab0572204e85 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0551df6be30c48698778ab0572204e85 2023-04-07 09:54:08,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3218114cbce54659ae3c5f30eb6d368a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3218114cbce54659ae3c5f30eb6d368a 2023-04-07 09:54:08,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/22e1716e99b4483b8c95f549c2bab3e3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/22e1716e99b4483b8c95f549c2bab3e3 2023-04-07 09:54:08,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2ba0150fcc654f41a99983e0779b0666 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/2ba0150fcc654f41a99983e0779b0666 2023-04-07 09:54:08,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91fdad2fce2a448b86d20cf8366d9df3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/91fdad2fce2a448b86d20cf8366d9df3 2023-04-07 09:54:08,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/4dd1b444e43043f7a9fbff6742f48138 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/4dd1b444e43043f7a9fbff6742f48138 2023-04-07 09:54:08,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/46889ae1a1914c5eb8368ef020a6131b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/46889ae1a1914c5eb8368ef020a6131b 2023-04-07 09:54:08,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/afa0b3b01620403aa9e5f4f8c2d3ca5b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/afa0b3b01620403aa9e5f4f8c2d3ca5b 2023-04-07 09:54:08,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/541d32ada9b44d5bb4f3b13ec8d05fa1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/541d32ada9b44d5bb4f3b13ec8d05fa1 2023-04-07 09:54:08,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3d6c2eb200044017b57d23956cc95928 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/3d6c2eb200044017b57d23956cc95928 2023-04-07 09:54:08,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d68989f856d941eda9dd5746b990bf41 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/d68989f856d941eda9dd5746b990bf41 2023-04-07 09:54:08,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c89d738c06b640e8bad8e1e7151ac716 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/c89d738c06b640e8bad8e1e7151ac716 2023-04-07 09:54:08,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0ebc0bed7b254759a3bf44635d12ca04 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/0ebc0bed7b254759a3bf44635d12ca04 2023-04-07 09:54:08,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/35fe938aa7904d3bacdeac19e5e627d3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/35fe938aa7904d3bacdeac19e5e627d3 2023-04-07 09:54:08,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bf96b3e5047442d48b76322df0193a4c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1c5838df8fd344e591b4ab74fe7ab390, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1d56fb8b0feb487fb24f18f91d21a9ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b7b6b22a7e864007b687972d5b38baca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/2cc01b012bdb424188b756480dd329e8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b2690cb370fd468b83c603646628b481, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/a4c24100c38d4e29b5d153f0073c7afa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bfb051901c9d4ac2a244175d94e45169, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/f88c33938b724287898eaf614709adc8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/12604177e5ac4487af4bdf23b0116492, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bb68bc95db2f4dde88ebfee99f1822b9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/0a5f83d7d46f43e09ae2299fd7927496, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ad6ffcab34649c9bf9254d4a93c2a60, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ce965889047f4d689b963337f4a7dcb3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4b4ec9799e0d42bf8ffbe4c921b4c13d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/60076e6079c24287b9de4f593e568288, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b51900de50944e6db09789419c19117f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4f0700d488f849468c8ca35572919c78, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/49bd425545e5463093f302f7fd1ece89, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/917decd7ffbb46b5ade856405e40c7c3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ed30df31f73b40058d5581dd7ddb21b2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b0aab677a4e04a8ab5d29960664809de, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4416dda0d1b744f68aa06c1122647d00, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/8d523b5835ff41cf8a9de904fcfe3ea2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/e98235fd952c4de0a5fabf5d3a375d28, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/06094dff34044f54b0a49939026893b8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1adc8b17adb3480aa2fba584413eb4c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ae263c51ab74c1788c2488b70225c77, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/99d1563357af47188101e27610f877f0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ab583a37c2740efb66beef5d2b9b56f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/66415c78c7c347c2af2f6b9747f40ac6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/78794d15ef3e417dbef7fd56607b8214, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7434c034fb83446b80e9c136324b29b7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/c1c74bf7716a4c59a497384a97763e9c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/549cfa8dd8074f6aabdd42314b005cd5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/daf1724b9a24480a920c37f588e34633, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/776e44fe5cc74bdcad81b64912eb7689, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7263772058ef4394922b70aef06e0857, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/00f7388a61c744678a9c03080375585e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/3e9c7c663ab54611a06235abf5af5c11, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5c720610e5a54ec494fb0568b93775af, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/d4457710e24f4c679ce20e3e09156cf4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/089f0a26e4e04860b1119999ed1aec59] to archive 2023-04-07 09:54:08,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:54:08,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bf96b3e5047442d48b76322df0193a4c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bf96b3e5047442d48b76322df0193a4c 2023-04-07 09:54:08,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1c5838df8fd344e591b4ab74fe7ab390 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1c5838df8fd344e591b4ab74fe7ab390 2023-04-07 09:54:08,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1d56fb8b0feb487fb24f18f91d21a9ed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1d56fb8b0feb487fb24f18f91d21a9ed 2023-04-07 09:54:08,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b7b6b22a7e864007b687972d5b38baca to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b7b6b22a7e864007b687972d5b38baca 2023-04-07 09:54:08,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/2cc01b012bdb424188b756480dd329e8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/2cc01b012bdb424188b756480dd329e8 2023-04-07 09:54:08,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b2690cb370fd468b83c603646628b481 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b2690cb370fd468b83c603646628b481 2023-04-07 09:54:08,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/a4c24100c38d4e29b5d153f0073c7afa to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/a4c24100c38d4e29b5d153f0073c7afa 2023-04-07 09:54:08,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bfb051901c9d4ac2a244175d94e45169 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bfb051901c9d4ac2a244175d94e45169 2023-04-07 09:54:08,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/f88c33938b724287898eaf614709adc8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/f88c33938b724287898eaf614709adc8 2023-04-07 09:54:08,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/12604177e5ac4487af4bdf23b0116492 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/12604177e5ac4487af4bdf23b0116492 2023-04-07 09:54:08,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bb68bc95db2f4dde88ebfee99f1822b9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/bb68bc95db2f4dde88ebfee99f1822b9 2023-04-07 09:54:08,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/0a5f83d7d46f43e09ae2299fd7927496 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/0a5f83d7d46f43e09ae2299fd7927496 2023-04-07 09:54:08,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ad6ffcab34649c9bf9254d4a93c2a60 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ad6ffcab34649c9bf9254d4a93c2a60 2023-04-07 09:54:08,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ce965889047f4d689b963337f4a7dcb3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ce965889047f4d689b963337f4a7dcb3 2023-04-07 09:54:08,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4b4ec9799e0d42bf8ffbe4c921b4c13d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4b4ec9799e0d42bf8ffbe4c921b4c13d 2023-04-07 09:54:08,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/60076e6079c24287b9de4f593e568288 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/60076e6079c24287b9de4f593e568288 2023-04-07 09:54:08,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b51900de50944e6db09789419c19117f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b51900de50944e6db09789419c19117f 2023-04-07 09:54:08,414 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4f0700d488f849468c8ca35572919c78 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4f0700d488f849468c8ca35572919c78 2023-04-07 09:54:08,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/49bd425545e5463093f302f7fd1ece89 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/49bd425545e5463093f302f7fd1ece89 2023-04-07 09:54:08,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/917decd7ffbb46b5ade856405e40c7c3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/917decd7ffbb46b5ade856405e40c7c3 2023-04-07 09:54:08,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ed30df31f73b40058d5581dd7ddb21b2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/ed30df31f73b40058d5581dd7ddb21b2 2023-04-07 09:54:08,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b0aab677a4e04a8ab5d29960664809de to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/b0aab677a4e04a8ab5d29960664809de 2023-04-07 09:54:08,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4416dda0d1b744f68aa06c1122647d00 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/4416dda0d1b744f68aa06c1122647d00 2023-04-07 09:54:08,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/8d523b5835ff41cf8a9de904fcfe3ea2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/8d523b5835ff41cf8a9de904fcfe3ea2 2023-04-07 09:54:08,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/e98235fd952c4de0a5fabf5d3a375d28 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/e98235fd952c4de0a5fabf5d3a375d28 2023-04-07 09:54:08,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/06094dff34044f54b0a49939026893b8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/06094dff34044f54b0a49939026893b8 2023-04-07 09:54:08,425 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1adc8b17adb3480aa2fba584413eb4c1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/1adc8b17adb3480aa2fba584413eb4c1 2023-04-07 09:54:08,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ae263c51ab74c1788c2488b70225c77 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ae263c51ab74c1788c2488b70225c77 2023-04-07 09:54:08,428 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/99d1563357af47188101e27610f877f0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/99d1563357af47188101e27610f877f0 2023-04-07 09:54:08,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ab583a37c2740efb66beef5d2b9b56f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5ab583a37c2740efb66beef5d2b9b56f 2023-04-07 09:54:08,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/66415c78c7c347c2af2f6b9747f40ac6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/66415c78c7c347c2af2f6b9747f40ac6 2023-04-07 09:54:08,431 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/78794d15ef3e417dbef7fd56607b8214 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/78794d15ef3e417dbef7fd56607b8214 2023-04-07 09:54:08,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7434c034fb83446b80e9c136324b29b7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7434c034fb83446b80e9c136324b29b7 2023-04-07 09:54:08,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/c1c74bf7716a4c59a497384a97763e9c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/c1c74bf7716a4c59a497384a97763e9c 2023-04-07 09:54:08,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/549cfa8dd8074f6aabdd42314b005cd5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/549cfa8dd8074f6aabdd42314b005cd5 2023-04-07 09:54:08,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/daf1724b9a24480a920c37f588e34633 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/daf1724b9a24480a920c37f588e34633 2023-04-07 09:54:08,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/776e44fe5cc74bdcad81b64912eb7689 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/776e44fe5cc74bdcad81b64912eb7689 2023-04-07 09:54:08,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7263772058ef4394922b70aef06e0857 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/7263772058ef4394922b70aef06e0857 2023-04-07 09:54:08,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/00f7388a61c744678a9c03080375585e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/00f7388a61c744678a9c03080375585e 2023-04-07 09:54:08,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/3e9c7c663ab54611a06235abf5af5c11 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/3e9c7c663ab54611a06235abf5af5c11 2023-04-07 09:54:08,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5c720610e5a54ec494fb0568b93775af to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/5c720610e5a54ec494fb0568b93775af 2023-04-07 09:54:08,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/d4457710e24f4c679ce20e3e09156cf4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/d4457710e24f4c679ce20e3e09156cf4 2023-04-07 09:54:08,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/089f0a26e4e04860b1119999ed1aec59 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/089f0a26e4e04860b1119999ed1aec59 2023-04-07 09:54:08,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/recovered.edits/846.seqid, newMaxSeqId=846, maxSeqId=1 2023-04-07 09:54:08,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b. 2023-04-07 09:54:08,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0b39eaaa997d54d62464302ce4e71f9b: 2023-04-07 09:54:08,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:08,455 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=0b39eaaa997d54d62464302ce4e71f9b, regionState=CLOSED 2023-04-07 09:54:08,455 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861248454"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861248454"}]},"ts":"1680861248454"} 2023-04-07 09:54:08,457 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-07 09:54:08,457 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 0b39eaaa997d54d62464302ce4e71f9b, server=jenkins-hbase4.apache.org,35881,1680861218288 in 2.4560 sec 2023-04-07 09:54:08,459 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-07 09:54:08,459 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0b39eaaa997d54d62464302ce4e71f9b, UNASSIGN in 2.4620 sec 2023-04-07 09:54:08,460 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861248460"}]},"ts":"1680861248460"} 2023-04-07 09:54:08,461 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 09:54:08,462 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 09:54:08,464 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4830 sec 2023-04-07 09:54:10,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-07 09:54:10,097 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-07 09:54:10,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 09:54:10,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:10,109 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:10,110 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:10,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-07 09:54:10,115 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:10,118 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/recovered.edits] 2023-04-07 09:54:10,120 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2ea4eab5fcdb41a99ea4df918a1b5e23 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/2ea4eab5fcdb41a99ea4df918a1b5e23 2023-04-07 09:54:10,121 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/98ef0fc99c374580ac2c623786ddd69a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/98ef0fc99c374580ac2c623786ddd69a 2023-04-07 09:54:10,123 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f4960f657e4f4fa5bf7b4b3a5215d96a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/A/f4960f657e4f4fa5bf7b4b3a5215d96a 2023-04-07 09:54:10,125 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/429172c23d984d3f82525e2a6521a6db to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/429172c23d984d3f82525e2a6521a6db 2023-04-07 09:54:10,126 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/559f9bfd61184449a414c1092e817ce9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/559f9bfd61184449a414c1092e817ce9 2023-04-07 09:54:10,127 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7bc48506bae84d2bbb1c1e6052fa77ce to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/B/7bc48506bae84d2bbb1c1e6052fa77ce 2023-04-07 09:54:10,129 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/354c4e682cb649eda4184491403ddd12 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/354c4e682cb649eda4184491403ddd12 2023-04-07 09:54:10,130 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/365d31fab67b44058f2e9a4b0be579dc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/365d31fab67b44058f2e9a4b0be579dc 2023-04-07 09:54:10,131 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/466b0d721aed432aa59cd68173cd7795 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/C/466b0d721aed432aa59cd68173cd7795 2023-04-07 09:54:10,134 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/recovered.edits/846.seqid to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b/recovered.edits/846.seqid 2023-04-07 09:54:10,134 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/0b39eaaa997d54d62464302ce4e71f9b 2023-04-07 09:54:10,134 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:54:10,137 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:10,159 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 09:54:10,203 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 09:54:10,204 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:10,204 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 09:54:10,205 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680861250205"}]},"ts":"9223372036854775807"} 2023-04-07 09:54:10,207 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 09:54:10,207 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0b39eaaa997d54d62464302ce4e71f9b, NAME => 'TestAcidGuarantees,,1680861223351.0b39eaaa997d54d62464302ce4e71f9b.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 09:54:10,207 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 09:54:10,207 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680861250207"}]},"ts":"9223372036854775807"} 2023-04-07 09:54:10,209 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 09:54:10,211 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:10,212 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 109 msec 2023-04-07 09:54:10,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-07 09:54:10,213 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-07 09:54:10,228 INFO [Listener at localhost/39511] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2e4fa55b-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2125063445_15 at /127.0.0.1:45872 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2e4fa55b-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2125063445_15 at /127.0.0.1:45930 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2e4fa55b-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2e4fa55b-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:35881-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=593 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=412 (was 317) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3022 (was 3609) 2023-04-07 09:54:10,240 INFO [Listener at localhost/39511] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=593, MaxFileDescriptor=60000, SystemLoadAverage=412, ProcessCount=169, AvailableMemoryMB=3021 2023-04-07 09:54:10,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:54:10,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:54:10,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:10,245 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 09:54:10,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-07 09:54:10,245 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 09:54:10,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 09:54:10,247 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:10,247 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 empty. 2023-04-07 09:54:10,248 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:10,248 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:54:10,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 09:54:10,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 09:54:10,656 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:54:10,658 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b48a920edd75a7abe23cbbe5f3bc5475, NAME => 'TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp 2023-04-07 09:54:10,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 09:54:11,064 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:54:11,064 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b48a920edd75a7abe23cbbe5f3bc5475, disabling compactions & flushes 2023-04-07 09:54:11,065 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,065 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,065 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. after waiting 0 ms 2023-04-07 09:54:11,065 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,065 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,065 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:11,067 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 09:54:11,068 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861251068"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861251068"}]},"ts":"1680861251068"} 2023-04-07 09:54:11,069 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 09:54:11,070 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 09:54:11,070 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861251070"}]},"ts":"1680861251070"} 2023-04-07 09:54:11,071 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 09:54:11,074 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, ASSIGN}] 2023-04-07 09:54:11,076 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, ASSIGN 2023-04-07 09:54:11,077 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=false 2023-04-07 09:54:11,228 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:11,228 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861251228"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861251228"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861251228"}]},"ts":"1680861251228"} 2023-04-07 09:54:11,230 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:54:11,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 09:54:11,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b48a920edd75a7abe23cbbe5f3bc5475, NAME => 'TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:54:11,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:54:11,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,388 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,389 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A 2023-04-07 09:54:11,389 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A 2023-04-07 09:54:11,390 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:11,390 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b48a920edd75a7abe23cbbe5f3bc5475 columnFamilyName A 2023-04-07 09:54:11,391 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(310): Store=b48a920edd75a7abe23cbbe5f3bc5475/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:11,391 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,392 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B 2023-04-07 09:54:11,392 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B 2023-04-07 09:54:11,392 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:11,392 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b48a920edd75a7abe23cbbe5f3bc5475 columnFamilyName B 2023-04-07 09:54:11,393 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(310): Store=b48a920edd75a7abe23cbbe5f3bc5475/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:11,393 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,394 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C 2023-04-07 09:54:11,394 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C 2023-04-07 09:54:11,394 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:11,395 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b48a920edd75a7abe23cbbe5f3bc5475 columnFamilyName C 2023-04-07 09:54:11,395 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(310): Store=b48a920edd75a7abe23cbbe5f3bc5475/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:11,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:54:11,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:11,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:54:11,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b48a920edd75a7abe23cbbe5f3bc5475; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73511575, jitterRate=0.09540782868862152}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:54:11,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:11,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., pid=18, masterSystemTime=1680861251382 2023-04-07 09:54:11,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:11,405 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:11,405 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861251405"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861251405"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861251405"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861251405"}]},"ts":"1680861251405"} 2023-04-07 09:54:11,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-07 09:54:11,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 in 176 msec 2023-04-07 09:54:11,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-07 09:54:11,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, ASSIGN in 334 msec 2023-04-07 09:54:11,411 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 09:54:11,411 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861251411"}]},"ts":"1680861251411"} 2023-04-07 09:54:11,412 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 09:54:11,415 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 09:54:11,417 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1730 sec 2023-04-07 09:54:12,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-07 09:54:12,351 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-07 09:54:12,352 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:12,357 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:12,359 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:12,364 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58718, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:12,372 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 09:54:12,375 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 09:54:12,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:54:12,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:54:12,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:12,430 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 09:54:12,431 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:54:12,431 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 09:54:12,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-07 09:54:12,460 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, REOPEN/MOVE}] 2023-04-07 09:54:12,461 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, REOPEN/MOVE 2023-04-07 09:54:12,462 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:12,462 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861252462"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861252462"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861252462"}]},"ts":"1680861252462"} 2023-04-07 09:54:12,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:54:12,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b48a920edd75a7abe23cbbe5f3bc5475, disabling compactions & flushes 2023-04-07 09:54:12,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. after waiting 0 ms 2023-04-07 09:54:12,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-07 09:54:12,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:12,622 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: b48a920edd75a7abe23cbbe5f3bc5475 to self. 2023-04-07 09:54:12,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,624 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=CLOSED 2023-04-07 09:54:12,624 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861252624"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861252624"}]},"ts":"1680861252624"} 2023-04-07 09:54:12,626 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-07 09:54:12,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 in 162 msec 2023-04-07 09:54:12,627 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=true 2023-04-07 09:54:12,778 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:12,778 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861252777"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861252777"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861252777"}]},"ts":"1680861252777"} 2023-04-07 09:54:12,780 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:54:12,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b48a920edd75a7abe23cbbe5f3bc5475, NAME => 'TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:54:12,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:54:12,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,938 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,939 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A 2023-04-07 09:54:12,939 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A 2023-04-07 09:54:12,939 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:12,942 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b48a920edd75a7abe23cbbe5f3bc5475 columnFamilyName A 2023-04-07 09:54:12,944 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(310): Store=b48a920edd75a7abe23cbbe5f3bc5475/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:12,945 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,946 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B 2023-04-07 09:54:12,946 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B 2023-04-07 09:54:12,946 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:12,946 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b48a920edd75a7abe23cbbe5f3bc5475 columnFamilyName B 2023-04-07 09:54:12,947 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(310): Store=b48a920edd75a7abe23cbbe5f3bc5475/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:12,947 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,948 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C 2023-04-07 09:54:12,948 DEBUG [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C 2023-04-07 09:54:12,948 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:12,948 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b48a920edd75a7abe23cbbe5f3bc5475 columnFamilyName C 2023-04-07 09:54:12,949 INFO [StoreOpener-b48a920edd75a7abe23cbbe5f3bc5475-1] regionserver.HStore(310): Store=b48a920edd75a7abe23cbbe5f3bc5475/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:12,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:54:12,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:12,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b48a920edd75a7abe23cbbe5f3bc5475; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74238219, jitterRate=0.10623566806316376}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:54:12,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:12,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., pid=23, masterSystemTime=1680861252931 2023-04-07 09:54:12,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,957 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:12,957 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:12,958 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861252957"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861252957"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861252957"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861252957"}]},"ts":"1680861252957"} 2023-04-07 09:54:12,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-07 09:54:12,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 in 179 msec 2023-04-07 09:54:12,962 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-07 09:54:12,962 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, REOPEN/MOVE in 501 msec 2023-04-07 09:54:12,964 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-07 09:54:12,964 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 521 msec 2023-04-07 09:54:12,966 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 577 msec 2023-04-07 09:54:12,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-07 09:54:12,975 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:12,981 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:12,982 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:12,993 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:12,993 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:12,998 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:12,999 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:13,005 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:13,006 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:13,009 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:13,010 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:13,017 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:13,017 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:13,023 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:13,024 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:13,028 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:13,028 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:13,031 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:13,036 DEBUG [hconnection-0x6ad3f2da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,036 DEBUG [hconnection-0x3eeda380-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,037 DEBUG [hconnection-0x7e508080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,037 DEBUG [hconnection-0x581efffa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,039 DEBUG [hconnection-0x7d754821-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,039 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,039 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58728, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,040 DEBUG [hconnection-0x368dbd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,042 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58754, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,043 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,044 DEBUG [hconnection-0x247c451c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,044 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58760, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,045 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,049 DEBUG [hconnection-0x72566018-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,049 DEBUG [hconnection-0x76850bb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:13,049 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58774, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,056 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,056 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:13,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:13,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:13,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:13,062 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:13,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:13,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:13,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:13,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,073 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:54:13,074 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,074 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:13,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:13,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:13,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:13,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:13,076 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:13,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:13,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:13,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:13,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:13,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:13,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:13,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:13,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:13,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:13,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,085 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,085 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c57b029[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:13,087 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:13,088 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,090 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:13,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:13,096 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:13,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:13,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:13,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:13,096 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:13,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861313094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861313096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861313098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861313099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861313103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:13,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:13,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:13,108 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:13,108 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:13,161 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407b2ec243b57544ab59b3ef2eb2058a3b1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b2ec243b57544ab59b3ef2eb2058a3b1 2023-04-07 09:54:13,191 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/1e687068c97f4b928536bbdc47819f32 2023-04-07 09:54:13,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861313200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861313200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861313203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861313204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861313205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,208 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:13,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:13,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:13,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:13,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:13,224 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:13,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:13,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:13,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,225 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:13,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:13,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:13,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:13,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:13,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:13,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:13,236 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,236 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:13,236 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:13,236 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:13,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:13,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,237 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:13,237 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:13,238 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@549aec57[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:13,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:13,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,242 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:13,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:13,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:13,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,247 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:13,250 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:13,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:13,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:13,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:13,250 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:13,252 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,252 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,252 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,257 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,257 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:13,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:13,258 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:13,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:13,258 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:13,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,358 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:13,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:13,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:13,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:13,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:13,368 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:13,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:13,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:13,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:13,372 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:13,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:13,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:13,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:13,376 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:13,376 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,376 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:13,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:13,384 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:13,384 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,384 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:13,384 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:13,384 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,384 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:13,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:13,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cd24a21[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:13,391 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:13,391 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,394 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,394 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:13,394 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:13,394 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:13,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:13,397 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:13,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:13,401 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:13,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:13,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:13,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:13,401 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:13,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861313402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,405 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861313403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,405 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861313406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,407 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861313408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861313409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,411 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,413 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:13,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,414 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,414 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:13,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:13,415 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:13,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,415 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:13,516 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:13,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:13,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:13,524 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:13,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:13,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:13,525 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,525 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,527 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:13,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:13,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:13,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:13,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,529 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:13,532 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,532 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,532 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:13,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:13,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:13,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d380b84[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:13,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,537 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:13,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:13,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:13,537 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,537 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:13,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:13,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:13,543 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:13,543 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:13,543 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:13,543 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:13,543 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:13,545 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,545 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,545 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,546 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,551 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,551 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:13,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:13,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:13,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,552 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:13,553 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:13,635 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/fded5e53e3ba4ffc8d0e0be03b0f4c53 2023-04-07 09:54:13,653 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:13,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:13,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:13,664 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:13,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:13,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:13,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/b934c1e3857a4b88a20e36e765562848 2023-04-07 09:54:13,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/1e687068c97f4b928536bbdc47819f32 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1e687068c97f4b928536bbdc47819f32 2023-04-07 09:54:13,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,675 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:13,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:13,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:13,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:13,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:13,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:13,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:13,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:13,679 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,679 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:13,679 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:13,679 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:13,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:13,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:13,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:13,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1e687068c97f4b928536bbdc47819f32, entries=150, sequenceid=16, filesize=25.1 K 2023-04-07 09:54:13,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:13,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/fded5e53e3ba4ffc8d0e0be03b0f4c53 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/fded5e53e3ba4ffc8d0e0be03b0f4c53 2023-04-07 09:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cbf32f2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:13,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:13,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,685 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:13,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:13,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:13,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:13,687 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:13,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/fded5e53e3ba4ffc8d0e0be03b0f4c53, entries=150, sequenceid=16, filesize=11.6 K 2023-04-07 09:54:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/b934c1e3857a4b88a20e36e765562848 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/b934c1e3857a4b88a20e36e765562848 2023-04-07 09:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:13,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:13,691 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:13,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:13,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:13,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:13,691 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:13,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/b934c1e3857a4b88a20e36e765562848, entries=150, sequenceid=16, filesize=11.6 K 2023-04-07 09:54:13,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b48a920edd75a7abe23cbbe5f3bc5475 in 625ms, sequenceid=16, compaction requested=false 2023-04-07 09:54:13,699 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-07 09:54:13,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:13,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:13,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:13,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,708 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:13,708 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:13,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:13,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861253093, max timestamp=1680861253710Num uniques 859; 2023-04-07 09:54:13,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:13,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861253093, max timestamp=1680861253710Num uniques 859; 2023-04-07 09:54:13,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:13,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861253093, max timestamp=1680861253710Num uniques 859; 2023-04-07 09:54:13,716 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-07 09:54:13,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:54:13,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:13,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:13,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:13,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:13,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:13,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:13,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861253731, max timestamp=1680861253757Num uniques 859; 2023-04-07 09:54:13,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:54:13,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861253738, max timestamp=1680861253757Num uniques 859; 2023-04-07 09:54:13,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:13,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861253738, max timestamp=1680861253757Num uniques 859; 2023-04-07 09:54:13,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:54:13,775 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407fdc2bd28db7649ae8ec61c1583fdab62 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fdc2bd28db7649ae8ec61c1583fdab62 2023-04-07 09:54:13,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861313772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861313773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861313780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861313783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861313786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,804 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=40, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/bbfae75e34bb47cbb12c225502957242 2023-04-07 09:54:13,808 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:13,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:13,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:13,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:13,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:13,844 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:13,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:13,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:13,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,846 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:13,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:13,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:13,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:13,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861313888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861313891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861313892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:13,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861313894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861313888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:13,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:13,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:13,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:13,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:13,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:13,936 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:13,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=40 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/f5b19a9257ff4249aa3c00ba4d8143f5 2023-04-07 09:54:13,938 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f448366[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:13,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:13,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:13,947 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:13,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,949 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:13,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:13,953 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:13,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:13,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:13,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:13,954 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:13,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:13,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:13,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:13,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:13,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:13,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:13,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:13,964 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:13,964 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:13,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:13,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:13,999 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 09:54:14,064 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:14,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:14,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:14,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:14,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:14,068 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:14,068 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,068 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:14,068 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:14,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,070 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,070 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,070 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,070 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:14,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:14,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:14,075 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,075 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:14,075 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:14,075 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:14,075 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:14,078 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41695fed[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:14,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:14,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,082 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:14,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:14,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,085 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:14,089 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:14,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:14,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:14,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:14,090 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:14,091 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,091 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861314093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861314094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:14,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:14,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,101 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,101 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,101 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861314098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,101 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:14,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861314099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,101 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,101 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:14,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861314105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,202 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:14,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:14,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:14,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:14,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:14,206 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:14,206 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,206 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:14,206 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:14,207 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,207 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,208 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,208 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,209 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,209 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:14,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:14,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:14,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:14,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:14,215 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,215 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:14,215 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:14,215 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:14,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:14,217 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,218 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c70186f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:14,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:14,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:14,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:14,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:14,223 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:14,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:14,226 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:14,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:14,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:14,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:14,226 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:14,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:14,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,234 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:14,234 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:14,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:14,334 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:14,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:14,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:14,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:14,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:14,338 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:14,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:14,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:14,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:14,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:14,342 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,342 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:14,342 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:14,342 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:14,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:14,343 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294d95b1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:14,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:14,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,346 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:14,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:14,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:14,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:14,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:14,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:14,350 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:14,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:14,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:14,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:14,350 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:14,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,353 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:14,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:14,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,357 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:14,357 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:14,357 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:14,357 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,373 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.26 KB at sequenceid=40 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/876c86b04a9648e0923d603e4a2238ed 2023-04-07 09:54:14,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/bbfae75e34bb47cbb12c225502957242 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/bbfae75e34bb47cbb12c225502957242 2023-04-07 09:54:14,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/bbfae75e34bb47cbb12c225502957242, entries=200, sequenceid=40, filesize=31.9 K 2023-04-07 09:54:14,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/f5b19a9257ff4249aa3c00ba4d8143f5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f5b19a9257ff4249aa3c00ba4d8143f5 2023-04-07 09:54:14,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f5b19a9257ff4249aa3c00ba4d8143f5, entries=150, sequenceid=40, filesize=11.6 K 2023-04-07 09:54:14,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/876c86b04a9648e0923d603e4a2238ed as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/876c86b04a9648e0923d603e4a2238ed 2023-04-07 09:54:14,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861314401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861314401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861314404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861314404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/876c86b04a9648e0923d603e4a2238ed, entries=150, sequenceid=40, filesize=11.6 K 2023-04-07 09:54:14,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.57 KB/47685, heapSize ~125.04 KB/128040, currentSize=140.49 KB/143865 for b48a920edd75a7abe23cbbe5f3bc5475 in 697ms, sequenceid=40, compaction requested=false 2023-04-07 09:54:14,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:14,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861253777, max timestamp=1680861254417Num uniques 1068; 2023-04-07 09:54:14,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:54:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861253777, max timestamp=1680861254417Num uniques 1118; 2023-04-07 09:54:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-07 09:54:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861253777, max timestamp=1680861254417Num uniques 1009; 2023-04-07 09:54:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-07 09:54:14,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:14,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-07 09:54:14,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:14,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:14,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:14,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:14,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:14,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:14,457 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:14,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:14,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:14,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:14,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:14,462 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:14,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:14,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:14,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,464 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:14,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:14,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:14,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:14,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:14,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:14,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:14,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:14,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:14,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d97c4d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:14,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:14,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:14,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:14,484 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407087011828c89427284b59babae9674e0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407087011828c89427284b59babae9674e0 2023-04-07 09:54:14,485 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:14,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:14,489 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:14,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:14,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:14,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:14,490 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:14,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,494 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/d8d808cdca1e479a84c546db1f766538 2023-04-07 09:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,496 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861254467, max timestamp=1680861254504Num uniques 859; 2023-04-07 09:54:14,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:54:14,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861254467, max timestamp=1680861254504Num uniques 859; 2023-04-07 09:54:14,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:54:14,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861254467, max timestamp=1680861254504Num uniques 859; 2023-04-07 09:54:14,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:54:14,507 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:14,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:14,508 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:14,508 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:14,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,509 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,509 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,508 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/b81e4b73edf34abba81b240e177aef77 2023-04-07 09:54:14,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861314562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,608 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:14,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:14,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:14,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:14,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:14,612 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:14,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:14,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:14,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,615 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:14,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:14,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:14,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:14,617 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,617 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:14,617 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:14,617 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:14,619 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:14,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6944c1cc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:14,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:14,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,622 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,622 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:14,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:14,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:14,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,624 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,624 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,624 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:14,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:14,627 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:14,627 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:14,627 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:14,627 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:14,627 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:14,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:14,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:14,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,645 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:14,645 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:14,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:14,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861314668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,745 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:14,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:14,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:14,751 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:14,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:14,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:14,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:14,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:14,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:14,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:14,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:14,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:14,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:14,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:14,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:14,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:14,767 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:14,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@554b0feb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:14,773 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:14,773 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,775 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:14,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:14,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:14,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:14,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:14,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:14,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:14,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:14,783 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:14,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:14,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:14,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:14,783 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:14,785 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,785 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:14,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:14,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,793 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:14,793 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:14,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:14,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861314870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,893 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:14,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:14,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:14,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:14,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:14,898 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:14,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:14,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:14,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,900 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,900 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:14,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:14,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:14,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:14,902 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,902 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:14,902 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:14,902 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:14,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:14,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10eef8ee[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:14,907 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:14,907 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861314908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,909 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:14,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:14,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:14,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:14,911 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,911 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,911 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,911 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861314910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861314910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861314912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:14,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:14,916 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:14,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:14,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:14,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:14,916 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:14,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:14,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:14,919 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:14,920 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,924 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:14,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:14,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:14,924 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:14,924 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:14,928 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:14,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:14,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:14,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a5cc80d888b94b3d860e3863b3ce358e 2023-04-07 09:54:14,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/d8d808cdca1e479a84c546db1f766538 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d8d808cdca1e479a84c546db1f766538 2023-04-07 09:54:14,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d8d808cdca1e479a84c546db1f766538, entries=150, sequenceid=82, filesize=25.1 K 2023-04-07 09:54:14,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/b81e4b73edf34abba81b240e177aef77 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/b81e4b73edf34abba81b240e177aef77 2023-04-07 09:54:14,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/b81e4b73edf34abba81b240e177aef77, entries=150, sequenceid=82, filesize=11.6 K 2023-04-07 09:54:14,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a5cc80d888b94b3d860e3863b3ce358e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5cc80d888b94b3d860e3863b3ce358e 2023-04-07 09:54:14,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5cc80d888b94b3d860e3863b3ce358e, entries=150, sequenceid=82, filesize=11.6 K 2023-04-07 09:54:14,979 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.03 KB/148512, currentSize=134.18 KB/137400 for b48a920edd75a7abe23cbbe5f3bc5475 in 544ms, sequenceid=82, compaction requested=true 2023-04-07 09:54:14,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:14,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:14,980 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:14,980 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:14,980 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:14,980 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:14,981 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:14,981 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:14,982 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:14,982 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,982 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1e687068c97f4b928536bbdc47819f32, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/bbfae75e34bb47cbb12c225502957242, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d8d808cdca1e479a84c546db1f766538] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=82.2 K 2023-04-07 09:54:14,982 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:14,982 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:14,982 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/fded5e53e3ba4ffc8d0e0be03b0f4c53, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f5b19a9257ff4249aa3c00ba4d8143f5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/b81e4b73edf34abba81b240e177aef77] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=34.8 K 2023-04-07 09:54:14,983 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1e687068c97f4b928536bbdc47819f32, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680861253063 2023-04-07 09:54:14,983 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bbfae75e34bb47cbb12c225502957242, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=40, earliestPutTs=1680861253094 2023-04-07 09:54:14,983 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fded5e53e3ba4ffc8d0e0be03b0f4c53, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680861253063 2023-04-07 09:54:14,984 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d8d808cdca1e479a84c546db1f766538, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680861254426 2023-04-07 09:54:14,984 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f5b19a9257ff4249aa3c00ba4d8143f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=40, earliestPutTs=1680861253712 2023-04-07 09:54:14,984 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b81e4b73edf34abba81b240e177aef77, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680861254426 2023-04-07 09:54:15,001 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#148 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:15,024 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:15,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:15,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:15,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:15,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:15,029 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:15,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:15,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:15,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:15,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,031 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#147 average throughput is 0.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:15,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:15,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:15,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:15,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:15,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:15,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:15,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:15,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:15,072 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:15,072 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:15,072 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-07 09:54:15,072 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:15,073 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:15,073 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:15,073 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:15,073 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:15,073 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:15,145 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407729e96151cce40bea09a3b155c3ea566 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407729e96151cce40bea09a3b155c3ea566 2023-04-07 09:54:15,169 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/4f746498e4cc410396b698032ca28b0f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4f746498e4cc410396b698032ca28b0f 2023-04-07 09:54:15,175 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/82b05ca10d2642a794640a83dd06f593 2023-04-07 09:54:15,175 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into 4f746498e4cc410396b698032ca28b0f(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:15,176 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:15,176 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=13, startTime=1680861254979; duration=0sec 2023-04-07 09:54:15,176 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:15,176 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:15,181 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:15,181 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:15,181 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:15,181 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/b934c1e3857a4b88a20e36e765562848, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/876c86b04a9648e0923d603e4a2238ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5cc80d888b94b3d860e3863b3ce358e] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=34.8 K 2023-04-07 09:54:15,182 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b934c1e3857a4b88a20e36e765562848, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680861253063 2023-04-07 09:54:15,183 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 876c86b04a9648e0923d603e4a2238ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=40, earliestPutTs=1680861253712 2023-04-07 09:54:15,184 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting a5cc80d888b94b3d860e3863b3ce358e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680861254426 2023-04-07 09:54:15,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:15,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:15,194 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/16255d2e05ef49dca3e95a2569569edc 2023-04-07 09:54:15,204 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#151 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:15,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861315208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,243 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/c7b01ee857314135a1410fa420b75d2b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c7b01ee857314135a1410fa420b75d2b 2023-04-07 09:54:15,256 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into c7b01ee857314135a1410fa420b75d2b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:15,256 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:15,256 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=13, startTime=1680861254980; duration=0sec 2023-04-07 09:54:15,256 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:15,265 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/0f1b6046071c42548fc4685e8f03607a 2023-04-07 09:54:15,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/82b05ca10d2642a794640a83dd06f593 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/82b05ca10d2642a794640a83dd06f593 2023-04-07 09:54:15,291 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/82b05ca10d2642a794640a83dd06f593, entries=150, sequenceid=117, filesize=25.1 K 2023-04-07 09:54:15,293 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/16255d2e05ef49dca3e95a2569569edc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/16255d2e05ef49dca3e95a2569569edc 2023-04-07 09:54:15,311 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/16255d2e05ef49dca3e95a2569569edc, entries=150, sequenceid=117, filesize=11.6 K 2023-04-07 09:54:15,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/0f1b6046071c42548fc4685e8f03607a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/0f1b6046071c42548fc4685e8f03607a 2023-04-07 09:54:15,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861315312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,349 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/0f1b6046071c42548fc4685e8f03607a, entries=150, sequenceid=117, filesize=11.6 K 2023-04-07 09:54:15,350 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for b48a920edd75a7abe23cbbe5f3bc5475 in 278ms, sequenceid=117, compaction requested=false 2023-04-07 09:54:15,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:15,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:15,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:15,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:15,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:15,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:15,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:15,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fcd5aa[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:15,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:15,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:15,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:15,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:15,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:15,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:15,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:15,360 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:15,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:15,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:15,366 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:15,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:15,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:15,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:15,366 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:15,369 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,369 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:15,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,369 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:15,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:15,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:15,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,377 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:15,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:15,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:15,377 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:15,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:15,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:15,377 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:15,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,378 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:15,378 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:15,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:15,465 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/90482451bee04a118800c90365298c83 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/90482451bee04a118800c90365298c83 2023-04-07 09:54:15,472 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into 90482451bee04a118800c90365298c83(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:15,472 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:15,472 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=13, startTime=1680861254980; duration=0sec 2023-04-07 09:54:15,472 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:15,478 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:15,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:15,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:15,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:15,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:15,484 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:15,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:15,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:15,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:15,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:15,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:15,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:15,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:15,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:15,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:15,489 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:15,489 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:15,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:15,535 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861255587, max timestamp=1680861255602Num uniques 859; 2023-04-07 09:54:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861255587, max timestamp=1680861255602Num uniques 859; 2023-04-07 09:54:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861255587, max timestamp=1680861255602Num uniques 859; 2023-04-07 09:54:15,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:15,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861315653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861315758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,914 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040701d97d82c51c4098a208d922ddb5c8f3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040701d97d82c51c4098a208d922ddb5c8f3 2023-04-07 09:54:15,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861315912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861315919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861315919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861315920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:15,921 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/f86bb541447042409b83de822db1157d 2023-04-07 09:54:15,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:15,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861315962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:16,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1680861316265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,333 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/3ca59e6a67dc46368a3b9417635f8e3d 2023-04-07 09:54:16,745 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/424c9057d7674898b8f787258e3a6bab 2023-04-07 09:54:16,750 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/f86bb541447042409b83de822db1157d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f86bb541447042409b83de822db1157d 2023-04-07 09:54:16,754 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f86bb541447042409b83de822db1157d, entries=150, sequenceid=130, filesize=25.1 K 2023-04-07 09:54:16,755 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/3ca59e6a67dc46368a3b9417635f8e3d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/3ca59e6a67dc46368a3b9417635f8e3d 2023-04-07 09:54:16,760 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/3ca59e6a67dc46368a3b9417635f8e3d, entries=150, sequenceid=130, filesize=11.6 K 2023-04-07 09:54:16,761 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/424c9057d7674898b8f787258e3a6bab as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/424c9057d7674898b8f787258e3a6bab 2023-04-07 09:54:16,765 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/424c9057d7674898b8f787258e3a6bab, entries=150, sequenceid=130, filesize=11.6 K 2023-04-07 09:54:16,766 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b48a920edd75a7abe23cbbe5f3bc5475 in 1277ms, sequenceid=130, compaction requested=true 2023-04-07 09:54:16,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:16,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:16,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:16,775 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:16,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:16,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-07 09:54:16,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:16,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:16,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:16,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:16,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:16,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@767939f8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:16,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:16,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,780 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:16,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:16,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:16,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:16,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:16,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:16,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:16,783 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:16,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:16,787 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:16,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:16,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:16,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:16,788 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:16,790 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,790 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:16,790 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:16,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:16,791 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,793 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:16,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:16,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,796 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040767dede6043ab4bf98e2d7c2692610627 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040767dede6043ab4bf98e2d7c2692610627 2023-04-07 09:54:16,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:16,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:16,796 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:16,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:16,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:16,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:16,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:16,797 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:16,797 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:16,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:16,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:16,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:16,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861316808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,897 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:16,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:16,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:16,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:16,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:16,901 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:16,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:16,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:16,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:16,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:16,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:16,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:16,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:16,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:16,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:16,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:16,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:16,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:16,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:16,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:16,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:16,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,906 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:16,906 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:16,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,907 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:16,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:16,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:16,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b40448b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:16,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:16,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,911 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:16,911 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:16,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:16,915 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:16,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:16,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861316912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:16,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:16,918 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:16,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:16,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:16,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:16,918 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:16,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:16,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:16,920 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:16,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:16,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:16,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:16,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:16,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:16,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:16,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,924 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:16,924 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:16,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:16,924 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:16,924 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:16,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:16,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:16,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,024 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,027 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,031 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,031 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,031 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,031 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,039 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c0efa85[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,054 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,061 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,061 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,063 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,063 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,063 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,064 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,067 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,074 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,074 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,075 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,076 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,076 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,076 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:17,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861317117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,176 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,182 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,186 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,186 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,188 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,188 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,188 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,188 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,189 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,189 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,190 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,190 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43803d2b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,197 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,197 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,197 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,199 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,199 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,201 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,201 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,201 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,201 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,202 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,203 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/860f1fb710554773a48fdd443e9a1b2b 2023-04-07 09:54:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,204 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,205 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,205 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,205 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:17,305 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,309 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,315 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c097a5a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,318 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,318 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,318 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,318 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,318 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,318 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,322 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,322 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,329 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,329 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,330 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,330 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:17,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861317424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,430 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,440 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,440 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,440 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,440 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,442 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,449 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,449 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,449 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5940d4c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,453 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,453 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,453 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,453 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,453 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,453 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,455 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,461 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,464 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,464 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:17,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,472 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,472 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,572 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,575 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,575 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,575 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,575 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,577 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,577 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,577 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,582 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea515a9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,584 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,584 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,585 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,585 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,586 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,589 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,589 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,602 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,602 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:17,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/bf45a48a3d2e4462b3641de592dab1c6 2023-04-07 09:54:17,703 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,706 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,707 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,707 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,711 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,711 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,713 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8e1433[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,713 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,715 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,718 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,720 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,720 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,722 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,726 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,726 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:17,826 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,831 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,836 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,838 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,840 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b626052[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,840 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,845 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,845 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,855 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,855 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:17,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861317920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,923 DEBUG [Thread-1010] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:54:17,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:17,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861317926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:17,932 DEBUG [Thread-1002] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:54:17,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861317929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,933 DEBUG [Thread-1004] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:54:17,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:17,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861317934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:17,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861317934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,937 DEBUG [Thread-1008] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:54:17,955 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:17,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:17,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:17,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:17,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:17,958 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:17,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:17,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:17,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,960 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:17,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:17,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:17,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:17,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:17,964 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:17,964 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:17,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:17,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:17,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:17,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:17,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,965 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,966 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,966 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:17,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eb9f01f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:17,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:17,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:17,970 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,970 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:17,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:17,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:17,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:17,973 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:17,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:17,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:17,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:17,973 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:17,975 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,975 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,975 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:17,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:17,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:17,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:17,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:17,988 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:17,989 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:17,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:17,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:18,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/416a67a1bdeb4d87861c06944a064709 2023-04-07 09:54:18,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/860f1fb710554773a48fdd443e9a1b2b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/860f1fb710554773a48fdd443e9a1b2b 2023-04-07 09:54:18,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/860f1fb710554773a48fdd443e9a1b2b, entries=200, sequenceid=168, filesize=32.1 K 2023-04-07 09:54:18,041 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/bf45a48a3d2e4462b3641de592dab1c6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bf45a48a3d2e4462b3641de592dab1c6 2023-04-07 09:54:18,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bf45a48a3d2e4462b3641de592dab1c6, entries=150, sequenceid=168, filesize=11.8 K 2023-04-07 09:54:18,048 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-07 09:54:18,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/416a67a1bdeb4d87861c06944a064709 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/416a67a1bdeb4d87861c06944a064709 2023-04-07 09:54:18,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/416a67a1bdeb4d87861c06944a064709, entries=150, sequenceid=168, filesize=11.8 K 2023-04-07 09:54:18,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for b48a920edd75a7abe23cbbe5f3bc5475 in 1280ms, sequenceid=168, compaction requested=true 2023-04-07 09:54:18,058 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:18,058 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:18,058 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:18,058 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:18,058 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:18,058 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:18,060 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110227 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:18,060 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:18,060 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:18,060 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4f746498e4cc410396b698032ca28b0f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/82b05ca10d2642a794640a83dd06f593, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f86bb541447042409b83de822db1157d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/860f1fb710554773a48fdd443e9a1b2b] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=107.6 K 2023-04-07 09:54:18,060 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:18,060 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:18,060 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:18,060 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4f746498e4cc410396b698032ca28b0f, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680861254426 2023-04-07 09:54:18,060 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/90482451bee04a118800c90365298c83, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/16255d2e05ef49dca3e95a2569569edc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/3ca59e6a67dc46368a3b9417635f8e3d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bf45a48a3d2e4462b3641de592dab1c6] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=46.7 K 2023-04-07 09:54:18,061 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 90482451bee04a118800c90365298c83, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680861254426 2023-04-07 09:54:18,062 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 82b05ca10d2642a794640a83dd06f593, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680861254539 2023-04-07 09:54:18,062 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 16255d2e05ef49dca3e95a2569569edc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680861254539 2023-04-07 09:54:18,062 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting f86bb541447042409b83de822db1157d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680861255189 2023-04-07 09:54:18,062 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3ca59e6a67dc46368a3b9417635f8e3d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680861255189 2023-04-07 09:54:18,062 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 860f1fb710554773a48fdd443e9a1b2b, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680861255640 2023-04-07 09:54:18,063 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting bf45a48a3d2e4462b3641de592dab1c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680861255640 2023-04-07 09:54:18,074 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#160 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:18,081 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#159 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:18,089 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:18,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:18,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:18,098 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/8dc910e183f448dd956525d0269597bb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/8dc910e183f448dd956525d0269597bb 2023-04-07 09:54:18,103 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into 8dc910e183f448dd956525d0269597bb(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:18,103 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:18,104 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=12, startTime=1680861258058; duration=0sec 2023-04-07 09:54:18,104 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:18,104 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:18,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:18,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:18,105 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:18,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:18,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:18,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:18,105 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:18,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:18,105 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:18,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:18,105 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:18,106 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c7b01ee857314135a1410fa420b75d2b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/0f1b6046071c42548fc4685e8f03607a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/424c9057d7674898b8f787258e3a6bab, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/416a67a1bdeb4d87861c06944a064709] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=46.7 K 2023-04-07 09:54:18,106 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c7b01ee857314135a1410fa420b75d2b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680861254426 2023-04-07 09:54:18,107 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0f1b6046071c42548fc4685e8f03607a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680861254539 2023-04-07 09:54:18,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:18,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:18,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:18,107 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 424c9057d7674898b8f787258e3a6bab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680861255189 2023-04-07 09:54:18,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:18,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:18,108 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 416a67a1bdeb4d87861c06944a064709, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680861255640 2023-04-07 09:54:18,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:18,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:18,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:18,132 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:18,132 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:18,132 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:18,136 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:18,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:18,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:18,136 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-07 09:54:18,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:18,136 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:18,137 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:18,137 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:18,137 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:18,137 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:18,139 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:18,157 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/4d80871f7afb450ea61078a46b87e354 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/4d80871f7afb450ea61078a46b87e354 2023-04-07 09:54:18,165 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into 4d80871f7afb450ea61078a46b87e354(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:18,165 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:18,165 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=12, startTime=1680861258058; duration=0sec 2023-04-07 09:54:18,165 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:18,170 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/17e40c4c53114611877fb6525cb9175d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/17e40c4c53114611877fb6525cb9175d 2023-04-07 09:54:18,182 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into 17e40c4c53114611877fb6525cb9175d(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:18,182 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:18,182 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=12, startTime=1680861258058; duration=0sec 2023-04-07 09:54:18,182 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:18,570 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304076c73be90670249798c5494e9a9e27df1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076c73be90670249798c5494e9a9e27df1 2023-04-07 09:54:18,577 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=178, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/f9cddf7436fb4cd3be0fd7aaa2803f6c 2023-04-07 09:54:18,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:18,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:18,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861258964, max timestamp=1680861258986Num uniques 859; 2023-04-07 09:54:18,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861258976, max timestamp=1680861258986Num uniques 859; 2023-04-07 09:54:18,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:18,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861258976, max timestamp=1680861258986Num uniques 859; 2023-04-07 09:54:18,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:18,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:19,004 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/a17f8b791f394cf2a3468377f18bb615 2023-04-07 09:54:19,016 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/6705cba5a2804ebcbc1c2e65204c7714 2023-04-07 09:54:19,022 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/f9cddf7436fb4cd3be0fd7aaa2803f6c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f9cddf7436fb4cd3be0fd7aaa2803f6c 2023-04-07 09:54:19,026 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f9cddf7436fb4cd3be0fd7aaa2803f6c, entries=150, sequenceid=178, filesize=25.3 K 2023-04-07 09:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/a17f8b791f394cf2a3468377f18bb615 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a17f8b791f394cf2a3468377f18bb615 2023-04-07 09:54:19,034 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a17f8b791f394cf2a3468377f18bb615, entries=150, sequenceid=178, filesize=11.8 K 2023-04-07 09:54:19,035 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/6705cba5a2804ebcbc1c2e65204c7714 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/6705cba5a2804ebcbc1c2e65204c7714 2023-04-07 09:54:19,042 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/6705cba5a2804ebcbc1c2e65204c7714, entries=150, sequenceid=178, filesize=11.8 K 2023-04-07 09:54:19,044 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=116.29 KB/119080 for b48a920edd75a7abe23cbbe5f3bc5475 in 908ms, sequenceid=178, compaction requested=false 2023-04-07 09:54:19,044 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:19,044 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:19,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=123.00 KB heapSize=350.22 KB 2023-04-07 09:54:19,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:19,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:19,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:19,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:19,046 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:19,046 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:19,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f7a23c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,054 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,059 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,059 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,064 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:19,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,069 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,071 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861319097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,171 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:19,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:19,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:19,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:19,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:19,175 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:19,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:19,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:19,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,176 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,176 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:19,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:19,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:19,178 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,178 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:19,179 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:19,179 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24840e9d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,191 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,191 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,193 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,195 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,196 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,197 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,197 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:19,197 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:19,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861319205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,297 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:19,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:19,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:19,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:19,301 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:19,301 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,301 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:19,301 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:19,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:19,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:19,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:19,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:19,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:19,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cec9aaf[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,313 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,313 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,314 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,314 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,314 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,315 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,315 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,316 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,316 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,320 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,320 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,322 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,322 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,334 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,334 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,334 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,334 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,334 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,335 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,335 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:19,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:19,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861319410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,435 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:19,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:19,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:19,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:19,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:19,439 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:19,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:19,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:19,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,440 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,440 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:19,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d24ae25[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,450 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,450 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,451 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,451 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,451 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,451 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,451 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,451 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,454 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,454 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,454 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,458 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,459 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,460 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,461 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,461 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,468 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,468 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,468 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,468 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,488 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e5dd8f663e7e48fb89ace33f42f7fb61 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e5dd8f663e7e48fb89ace33f42f7fb61 2023-04-07 09:54:19,569 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:19,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:19,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:19,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:19,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:19,572 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:19,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:19,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:19,573 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,573 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:19,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:19,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:19,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:19,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:19,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,579 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@180711ff[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,581 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,581 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,585 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,585 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,587 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,587 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,587 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,594 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,597 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:19,597 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,698 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:19,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:19,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:19,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:19,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:19,701 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:19,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:19,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:19,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,703 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:19,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:19,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:19,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:19,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf30484[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,711 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:19,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861319715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,718 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,718 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:19,724 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,724 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,824 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:19,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:19,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:19,828 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:19,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:19,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:19,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:19,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:19,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:19,831 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,832 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:19,832 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:19,832 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5482dc57[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,846 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,846 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,847 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,854 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,854 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,898 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=215, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/2b8f76d02cb04949b74d28e62f764698 2023-04-07 09:54:19,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/2fae2b6287924144b85606324c0dcaf5 2023-04-07 09:54:19,954 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:19,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:19,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:19,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:19,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:19,957 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:19,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:19,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:19,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,959 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:19,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:19,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:19,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:19,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:19,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2f0564[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:19,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:19,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:19,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:19,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:19,972 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:19,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:19,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:19,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:19,972 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:19,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:19,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,975 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:19,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:19,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:19,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:19,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:19,979 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:19,979 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:19,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:19,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:19,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:20,079 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:20,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:20,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:20,082 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:20,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:20,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:20,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,084 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:20,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:20,085 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,085 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:20,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:20,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:20,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:20,086 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,086 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:20,087 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:20,087 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:20,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:20,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:20,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:20,089 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c8771f7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:20,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:20,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:20,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:20,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:20,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:20,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:20,097 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:20,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:20,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:20,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:20,097 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:20,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:20,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,102 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,103 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,103 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:20,103 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:20,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:20,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:20,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:20,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:20,104 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:20,104 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:20,204 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:20,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:20,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:20,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:20,207 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:20,207 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,207 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:20,207 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:20,208 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,208 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,209 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,209 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:20,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:20,209 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,209 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:20,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:20,211 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:20,212 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,212 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:20,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d493c7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:20,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,219 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:20,219 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:20,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:20,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:20,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:20,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861320221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:20,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:20,223 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:20,223 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:20,223 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:20,223 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:20,223 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:20,224 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,224 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,224 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:20,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:20,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:20,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:20,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:20,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:20,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:20,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,230 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:20,230 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:20,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:20,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:20,330 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:20,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:20,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:20,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:20,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:20,333 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:20,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:20,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:20,334 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,334 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:20,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:20,336 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,336 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:20,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:20,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,338 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:20,338 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:20,338 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:20,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:20,339 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,339 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af696ee[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:20,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:20,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,342 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:20,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:20,345 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/8379fe636cb14d8cb808424b6f355d27 2023-04-07 09:54:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:20,349 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:20,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:20,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:20,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:20,349 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:20,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:20,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:20,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/2b8f76d02cb04949b74d28e62f764698 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2b8f76d02cb04949b74d28e62f764698 2023-04-07 09:54:20,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,353 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:20,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2b8f76d02cb04949b74d28e62f764698, entries=200, sequenceid=215, filesize=32.1 K 2023-04-07 09:54:20,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/2fae2b6287924144b85606324c0dcaf5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/2fae2b6287924144b85606324c0dcaf5 2023-04-07 09:54:20,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:20,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:20,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:20,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:20,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:20,361 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:20,361 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/2fae2b6287924144b85606324c0dcaf5, entries=150, sequenceid=215, filesize=11.8 K 2023-04-07 09:54:20,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/8379fe636cb14d8cb808424b6f355d27 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/8379fe636cb14d8cb808424b6f355d27 2023-04-07 09:54:20,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/8379fe636cb14d8cb808424b6f355d27, entries=150, sequenceid=215, filesize=11.8 K 2023-04-07 09:54:20,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~350.17 KB/358576, currentSize=60.38 KB/61830 for b48a920edd75a7abe23cbbe5f3bc5475 in 1331ms, sequenceid=215, compaction requested=true 2023-04-07 09:54:20,377 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:20,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:20,377 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:20,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:20,377 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:20,377 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:20,378 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:20,378 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:20,378 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,378 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/17e40c4c53114611877fb6525cb9175d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f9cddf7436fb4cd3be0fd7aaa2803f6c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2b8f76d02cb04949b74d28e62f764698] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=82.9 K 2023-04-07 09:54:20,379 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:20,379 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:20,379 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,379 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 17e40c4c53114611877fb6525cb9175d, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680861255640 2023-04-07 09:54:20,379 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/8dc910e183f448dd956525d0269597bb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a17f8b791f394cf2a3468377f18bb615, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/2fae2b6287924144b85606324c0dcaf5] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=35.5 K 2023-04-07 09:54:20,379 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting f9cddf7436fb4cd3be0fd7aaa2803f6c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680861256790 2023-04-07 09:54:20,379 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8dc910e183f448dd956525d0269597bb, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680861255640 2023-04-07 09:54:20,380 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 2b8f76d02cb04949b74d28e62f764698, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680861259024 2023-04-07 09:54:20,380 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a17f8b791f394cf2a3468377f18bb615, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680861256790 2023-04-07 09:54:20,380 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2fae2b6287924144b85606324c0dcaf5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680861259024 2023-04-07 09:54:20,389 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:20,399 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#168 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:20,461 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:20,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:20,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:20,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:20,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:20,469 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:20,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:20,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:20,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:20,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:20,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:20,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:20,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:20,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:20,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:20,475 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,475 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:20,475 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:54:20,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:20,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:20,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:20,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:20,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:20,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:20,504 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a1e2acd689a143c1be0744ccd154228d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a1e2acd689a143c1be0744ccd154228d 2023-04-07 09:54:20,807 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/4bf169503b724dfb94723f502ae27eff as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4bf169503b724dfb94723f502ae27eff 2023-04-07 09:54:20,814 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into 4bf169503b724dfb94723f502ae27eff(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:20,814 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:20,814 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=13, startTime=1680861260377; duration=0sec 2023-04-07 09:54:20,814 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:20,814 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:20,817 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:20,817 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:20,817 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:20,817 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/4d80871f7afb450ea61078a46b87e354, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/6705cba5a2804ebcbc1c2e65204c7714, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/8379fe636cb14d8cb808424b6f355d27] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=35.5 K 2023-04-07 09:54:20,818 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4d80871f7afb450ea61078a46b87e354, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680861255640 2023-04-07 09:54:20,818 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 6705cba5a2804ebcbc1c2e65204c7714, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1680861256790 2023-04-07 09:54:20,819 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8379fe636cb14d8cb808424b6f355d27, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680861259024 2023-04-07 09:54:20,826 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#171 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:20,837 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/3b92f5a8e7ac496593851640016986dc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/3b92f5a8e7ac496593851640016986dc 2023-04-07 09:54:20,857 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into 3b92f5a8e7ac496593851640016986dc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:20,857 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:20,857 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=13, startTime=1680861260377; duration=0sec 2023-04-07 09:54:20,857 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:20,911 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=227, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/b820a6d2015948ccb345690b951230a3 2023-04-07 09:54:20,930 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/9e39fa550672497b8201bf13cb484b48 2023-04-07 09:54:21,227 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/e5f6173dbfe34a4e9f377212825baf4d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e5f6173dbfe34a4e9f377212825baf4d 2023-04-07 09:54:21,233 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into e5f6173dbfe34a4e9f377212825baf4d(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:21,233 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:21,233 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=13, startTime=1680861260377; duration=0sec 2023-04-07 09:54:21,233 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:21,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:21,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:21,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:21,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861321339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,355 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/c3472f1e8a6d4608a8aeaadfdabb2cd2 2023-04-07 09:54:21,360 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/b820a6d2015948ccb345690b951230a3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/b820a6d2015948ccb345690b951230a3 2023-04-07 09:54:21,366 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/b820a6d2015948ccb345690b951230a3, entries=100, sequenceid=227, filesize=18.5 K 2023-04-07 09:54:21,369 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/9e39fa550672497b8201bf13cb484b48 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/9e39fa550672497b8201bf13cb484b48 2023-04-07 09:54:21,374 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/9e39fa550672497b8201bf13cb484b48, entries=100, sequenceid=227, filesize=9.4 K 2023-04-07 09:54:21,376 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/c3472f1e8a6d4608a8aeaadfdabb2cd2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c3472f1e8a6d4608a8aeaadfdabb2cd2 2023-04-07 09:54:21,381 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c3472f1e8a6d4608a8aeaadfdabb2cd2, entries=100, sequenceid=227, filesize=9.4 K 2023-04-07 09:54:21,382 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b48a920edd75a7abe23cbbe5f3bc5475 in 907ms, sequenceid=227, compaction requested=false 2023-04-07 09:54:21,382 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:21,382 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e26bef1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:21,386 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:21,386 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:21,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:21,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:21,395 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:21,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:21,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:21,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:21,396 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:21,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,405 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:21,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:21,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,406 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:21,406 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:21,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:21,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861261317, max timestamp=1680861261441Num uniques 859; 2023-04-07 09:54:21,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:21,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861261317, max timestamp=1680861261441Num uniques 859; 2023-04-07 09:54:21,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:21,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861261317, max timestamp=1680861261441Num uniques 859; 2023-04-07 09:54:21,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:21,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-04-07 09:54:21,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:21,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:21,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:21,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:21,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:21,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:21,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861261485, max timestamp=1680861261494Num uniques 859; 2023-04-07 09:54:21,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861261485, max timestamp=1680861261494Num uniques 859; 2023-04-07 09:54:21,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:21,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:21,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861261485, max timestamp=1680861261494Num uniques 859; 2023-04-07 09:54:21,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:21,506 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:21,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:21,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:21,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:21,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:21,511 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:21,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:21,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:21,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,513 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:21,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:21,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:21,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:21,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:21,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:21,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:21,519 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,519 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,520 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,520 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29996c0b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:21,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:21,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,523 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:21,523 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,523 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,525 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:21,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:21,529 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:21,529 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:21,529 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:21,529 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:21,529 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:21,533 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,533 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,534 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:21,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:21,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,540 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:21,540 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:21,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:21,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861261539, max timestamp=1680861261564Num uniques 1018; 2023-04-07 09:54:21,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:21,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861261539, max timestamp=1680861261564Num uniques 1018; 2023-04-07 09:54:21,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861261539, max timestamp=1680861261564Num uniques 1018; 2023-04-07 09:54:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:21,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861261620, max timestamp=1680861261636Num uniques 1027; 2023-04-07 09:54:21,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861261620, max timestamp=1680861261636Num uniques 1027; 2023-04-07 09:54:21,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861261620, max timestamp=1680861261636Num uniques 1027; 2023-04-07 09:54:21,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:21,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:21,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:21,640 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:21,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:21,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:21,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:21,644 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:21,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:21,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:21,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:21,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:21,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:21,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:21,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:21,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:21,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:21,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:21,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f8854[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:21,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,658 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:21,660 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:21,663 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:21,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:21,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:21,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:21,663 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:21,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,668 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:21,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:21,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,672 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:21,672 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:21,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680861261684, max timestamp=1680861261694Num uniques 1086; 2023-04-07 09:54:21,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 09:54:21,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680861261684, max timestamp=1680861261694Num uniques 1086; 2023-04-07 09:54:21,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 09:54:21,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680861261684, max timestamp=1680861261694Num uniques 1086; 2023-04-07 09:54:21,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 09:54:21,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861261732, max timestamp=1680861261759Num uniques 1045; 2023-04-07 09:54:21,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-07 09:54:21,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861261732, max timestamp=1680861261759Num uniques 1045; 2023-04-07 09:54:21,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861261732, max timestamp=1680861261759Num uniques 1045; 2023-04-07 09:54:21,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-07 09:54:21,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-07 09:54:21,772 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:21,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:21,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:21,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:21,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:21,777 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:21,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:21,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:21,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:21,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:21,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:21,796 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,798 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:21,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc03ff3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:21,801 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:21,801 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:21,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:21,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:21,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:21,811 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:21,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:21,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:21,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:21,811 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:21,814 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,814 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,814 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,819 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,819 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,819 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:21,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,820 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:21,820 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:21,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861261820, max timestamp=1680861261833Num uniques 1154; 2023-04-07 09:54:21,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-07 09:54:21,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861261820, max timestamp=1680861261833Num uniques 1154; 2023-04-07 09:54:21,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-07 09:54:21,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861261820, max timestamp=1680861261833Num uniques 1154; 2023-04-07 09:54:21,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=6870, suffix heap size=147984,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=150 2023-04-07 09:54:21,883 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304070a23d0d4fc164699b884d6ece9475aee to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070a23d0d4fc164699b884d6ece9475aee 2023-04-07 09:54:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680861261872, max timestamp=1680861261898Num uniques 1013; 2023-04-07 09:54:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 09:54:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680861261872, max timestamp=1680861261898Num uniques 1013; 2023-04-07 09:54:21,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 09:54:21,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680861261872, max timestamp=1680861261898Num uniques 1009; 2023-04-07 09:54:21,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=7458, suffix heap size=132904,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=163 2023-04-07 09:54:21,905 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/de775faaa4cb41d2b49ed043c625069c 2023-04-07 09:54:21,920 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:21,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:21,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:21,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:21,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:21,936 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:21,936 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,936 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:21,936 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:21,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,939 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,939 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,939 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:21,939 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:21,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:21,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:21,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:21,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:21,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:21,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:21,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:21,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,949 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:21,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50bad74c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:21,950 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:21,951 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,952 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:21,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:21,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:21,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:21,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:21,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:21,956 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:21,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:21,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:21,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:21,956 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:21,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:21,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14734, totalHeapSize=33728, min timestamp=1680861261908, max timestamp=1680861261959Num uniques 1022; 2023-04-07 09:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1680861261908, max timestamp=1680861261959Num uniques 1022; 2023-04-07 09:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46794, new segment data size=14734, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-04-07 09:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=14742, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-04-07 09:54:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14737, totalHeapSize=33728, min timestamp=1680861261908, max timestamp=1680861261959Num uniques 1022; 2023-04-07 09:54:21,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46797, new segment data size=14737, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-04-07 09:54:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:21,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:21,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:21,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:21,972 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:21,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:21,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:21,972 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,972 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:21,972 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:21,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:21,972 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:21,972 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:21,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:21,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861321983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:21,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861321984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:21,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=531, cellSize=24316, totalHeapSize=55464, min timestamp=1680861261967, max timestamp=1680861261987Num uniques 1181; 2023-04-07 09:54:21,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=24316, suffix heap size=150792,new segment heap size=55464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=531 2023-04-07 09:54:21,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861261977, max timestamp=1680861261987Num uniques 1181; 2023-04-07 09:54:21,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-04-07 09:54:21,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861261977, max timestamp=1680861261987Num uniques 1181; 2023-04-07 09:54:21,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-04-07 09:54:22,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861322062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680861262050, max timestamp=1680861262064Num uniques 1390; 2023-04-07 09:54:22,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=63660, new segment data size=15570, suffix heap size=172528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1390, new segment cells count=340 2023-04-07 09:54:22,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680861262050, max timestamp=1680861262064Num uniques 1140; 2023-04-07 09:54:22,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1680861262050, max timestamp=1680861262064Num uniques 1140; 2023-04-07 09:54:22,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-07 09:54:22,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-04-07 09:54:22,072 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:22,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:22,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:22,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:22,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:22,077 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:22,077 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,077 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:22,077 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:22,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,079 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,079 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,079 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:22,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:22,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:22,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:22,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:22,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:22,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,095 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:22,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a10064[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:22,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:22,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:22,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:22,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:22,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:22,103 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:22,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:22,106 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:22,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:22,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:22,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:22,106 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:22,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1680861262096, max timestamp=1680861262108Num uniques 1199; 2023-04-07 09:54:22,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54904, new segment data size=11394, suffix heap size=152664,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=249 2023-04-07 09:54:22,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11394, totalHeapSize=26136, min timestamp=1680861262096, max timestamp=1680861262108Num uniques 1199; 2023-04-07 09:54:22,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54904, new segment data size=11394, suffix heap size=152664,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=249 2023-04-07 09:54:22,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,114 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:22,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:22,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680861262096, max timestamp=1680861262108Num uniques 1199; 2023-04-07 09:54:22,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=11404, suffix heap size=152664,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=249 2023-04-07 09:54:22,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,115 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:22,115 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:22,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680861262128, max timestamp=1680861262136Num uniques 1108; 2023-04-07 09:54:22,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-07 09:54:22,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680861262128, max timestamp=1680861262136Num uniques 1108; 2023-04-07 09:54:22,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-07 09:54:22,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680861262128, max timestamp=1680861262136Num uniques 1108; 2023-04-07 09:54:22,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-07 09:54:22,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680861262155, max timestamp=1680861262160Num uniques 1067; 2023-04-07 09:54:22,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 09:54:22,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=517, cellSize=23672, totalHeapSize=54008, min timestamp=1680861262146, max timestamp=1680861262160Num uniques 1067; 2023-04-07 09:54:22,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=23672, suffix heap size=138936,new segment heap size=54008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=517 2023-04-07 09:54:22,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680861262155, max timestamp=1680861262160Num uniques 1067; 2023-04-07 09:54:22,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-07 09:54:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861322177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680861262171, max timestamp=1680861262180Num uniques 1076; 2023-04-07 09:54:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680861262171, max timestamp=1680861262180Num uniques 1376; 2023-04-07 09:54:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680861262171, max timestamp=1680861262180Num uniques 1076; 2023-04-07 09:54:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=17216, suffix heap size=139872,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=376 2023-04-07 09:54:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=17216, suffix heap size=139872,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=376 2023-04-07 09:54:22,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=63016, new segment data size=17216, suffix heap size=171072,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1376, new segment cells count=376 2023-04-07 09:54:22,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861322202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861322203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861322204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680861262194, max timestamp=1680861262204Num uniques 1235; 2023-04-07 09:54:22,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56560, new segment data size=13050, suffix heap size=156408,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=285 2023-04-07 09:54:22,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680861262194, max timestamp=1680861262204Num uniques 1235; 2023-04-07 09:54:22,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56560, new segment data size=13050, suffix heap size=156408,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=285 2023-04-07 09:54:22,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680861262194, max timestamp=1680861262204Num uniques 1235; 2023-04-07 09:54:22,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56560, new segment data size=13050, suffix heap size=156408,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=285 2023-04-07 09:54:22,215 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:22,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:22,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:22,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:22,220 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:22,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:22,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:22,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:22,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:22,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:22,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:22,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:22,232 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:22,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be5134c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:22,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:22,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,237 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:22,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:22,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:22,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:22,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:22,242 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:22,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:22,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:22,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:22,242 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:22,244 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,258 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,258 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,258 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,258 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,258 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:22,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:22,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:22,258 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,258 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,258 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,259 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,259 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,259 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:22,259 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680861262253, max timestamp=1680861262286Num uniques 1144; 2023-04-07 09:54:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-07 09:54:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680861262253, max timestamp=1680861262286Num uniques 1144; 2023-04-07 09:54:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-07 09:54:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680861262253, max timestamp=1680861262286Num uniques 1144; 2023-04-07 09:54:22,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=13464, suffix heap size=146944,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=294 2023-04-07 09:54:22,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/35864ff4f2024000b5760ee88ccdcab2 2023-04-07 09:54:22,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1680861262305, max timestamp=1680861262340Num uniques 1153; 2023-04-07 09:54:22,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=16165, suffix heap size=147880,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=353 2023-04-07 09:54:22,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1680861262305, max timestamp=1680861262340Num uniques 1153; 2023-04-07 09:54:22,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=16165, suffix heap size=147880,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=353 2023-04-07 09:54:22,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=353, cellSize=16165, totalHeapSize=36952, min timestamp=1680861262305, max timestamp=1680861262340Num uniques 1153; 2023-04-07 09:54:22,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=16165, suffix heap size=147880,new segment heap size=36952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=353 2023-04-07 09:54:22,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/efa25e40186a479a90748cf202c0b3de 2023-04-07 09:54:22,359 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:22,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:22,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/de775faaa4cb41d2b49ed043c625069c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/de775faaa4cb41d2b49ed043c625069c 2023-04-07 09:54:22,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:22,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/de775faaa4cb41d2b49ed043c625069c, entries=150, sequenceid=251, filesize=25.3 K 2023-04-07 09:54:22,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:22,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:22,368 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:22,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:22,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:22,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/35864ff4f2024000b5760ee88ccdcab2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/35864ff4f2024000b5760ee88ccdcab2 2023-04-07 09:54:22,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861322364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861322365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861322366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861322366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:22,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:22,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:22,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861322370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:22,375 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,375 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:22,375 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:22,375 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/35864ff4f2024000b5760ee88ccdcab2, entries=150, sequenceid=251, filesize=11.8 K 2023-04-07 09:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:22,376 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/efa25e40186a479a90748cf202c0b3de as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/efa25e40186a479a90748cf202c0b3de 2023-04-07 09:54:22,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/efa25e40186a479a90748cf202c0b3de, entries=150, sequenceid=251, filesize=11.8 K 2023-04-07 09:54:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=161.02 KB/164880 for b48a920edd75a7abe23cbbe5f3bc5475 in 944ms, sequenceid=251, compaction requested=true 2023-04-07 09:54:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,390 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:22,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:22,391 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:22,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31a08264[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:22,391 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:22,391 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:22,391 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:22,393 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:22,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:22,393 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71032 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:22,394 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:22,394 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,394 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e5f6173dbfe34a4e9f377212825baf4d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/b820a6d2015948ccb345690b951230a3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/de775faaa4cb41d2b49ed043c625069c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=69.4 K 2023-04-07 09:54:22,394 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:22,395 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:22,395 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,395 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4bf169503b724dfb94723f502ae27eff, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/9e39fa550672497b8201bf13cb484b48, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/35864ff4f2024000b5760ee88ccdcab2] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=33.3 K 2023-04-07 09:54:22,396 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e5f6173dbfe34a4e9f377212825baf4d, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680861259024 2023-04-07 09:54:22,396 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4bf169503b724dfb94723f502ae27eff, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680861259024 2023-04-07 09:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:22,399 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,399 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b820a6d2015948ccb345690b951230a3, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680861259091 2023-04-07 09:54:22,399 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9e39fa550672497b8201bf13cb484b48, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680861259091 2023-04-07 09:54:22,401 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 35864ff4f2024000b5760ee88ccdcab2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680861261335 2023-04-07 09:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,404 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting de775faaa4cb41d2b49ed043c625069c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680861261335 2023-04-07 09:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:22,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:22,409 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:22,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:22,410 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:22,412 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,412 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,412 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,413 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,420 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#177 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:22,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:22,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:22,424 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,425 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:22,425 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:22,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,431 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#178 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:22,439 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/658ac96170734bbeaab79d378ec2b93a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/658ac96170734bbeaab79d378ec2b93a 2023-04-07 09:54:22,482 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into 658ac96170734bbeaab79d378ec2b93a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:22,482 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:22,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:22,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=174.43 KB heapSize=487.03 KB 2023-04-07 09:54:22,482 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=13, startTime=1680861262391; duration=0sec 2023-04-07 09:54:22,482 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:22,483 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:22,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:22,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:22,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:22,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:22,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:22,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:22,492 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:22,492 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:22,492 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,492 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/3b92f5a8e7ac496593851640016986dc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c3472f1e8a6d4608a8aeaadfdabb2cd2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/efa25e40186a479a90748cf202c0b3de] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=33.3 K 2023-04-07 09:54:22,493 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3b92f5a8e7ac496593851640016986dc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1680861259024 2023-04-07 09:54:22,494 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c3472f1e8a6d4608a8aeaadfdabb2cd2, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680861259091 2023-04-07 09:54:22,494 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting efa25e40186a479a90748cf202c0b3de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680861261335 2023-04-07 09:54:22,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861322482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861322482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861322486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861322486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861322483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,525 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:22,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:22,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:22,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:22,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:22,532 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:22,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:22,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:22,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,533 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#180 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:22,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,535 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:22,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:22,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:22,540 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,540 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:22,541 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:22,541 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:22,541 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:22,542 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/c6512d87ef104798b247e7c69f96e5c1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/c6512d87ef104798b247e7c69f96e5c1 2023-04-07 09:54:22,543 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a633ab[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:22,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:22,547 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040792421baa8c9945f2875cf3f7f286bf9a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040792421baa8c9945f2875cf3f7f286bf9a 2023-04-07 09:54:22,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:22,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:22,552 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into c6512d87ef104798b247e7c69f96e5c1(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:22,552 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:22,552 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=13, startTime=1680861262391; duration=0sec 2023-04-07 09:54:22,552 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:22,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,562 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/31b552e992c04a1e801d8cd65e5987cf as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/31b552e992c04a1e801d8cd65e5987cf 2023-04-07 09:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:22,564 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:22,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:22,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:22,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:22,564 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:22,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,569 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into 31b552e992c04a1e801d8cd65e5987cf(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:22,569 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:22,569 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=13, startTime=1680861262391; duration=0sec 2023-04-07 09:54:22,569 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:22,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:22,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,574 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:22,574 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,574 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,575 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861322597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861322597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861322597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861322598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861322601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,674 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:22,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:22,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:22,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:22,678 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:22,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:22,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:22,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,680 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:22,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:22,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:22,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:22,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:22,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:22,683 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:22,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:22,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:22,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:22,685 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3eb71b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:22,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:22,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,690 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,690 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,690 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:22,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:22,694 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:22,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:22,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:22,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:22,694 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:22,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:22,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,700 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,700 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:22,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:22,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,800 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:22,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:22,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:22,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861322802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861322802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:22,804 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:22,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861322803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:22,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:22,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861322803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:22,805 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861322803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,805 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:22,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:22,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:22,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:22,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:22,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:22,809 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,809 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:22,809 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:22,809 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,809 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:22,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:22,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:22,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:22,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:22,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e22ba83[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:22,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:22,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,814 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:22,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:22,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:22,814 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,814 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:22,815 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:22,818 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:22,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:22,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:22,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:22,818 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:22,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,820 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,823 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:22,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,825 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:22,825 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:22,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,925 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:22,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:22,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:22,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:22,929 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:22,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:22,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:22,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:22,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:22,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:22,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:22,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:22,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:22,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:22,938 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,938 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:22,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba4a681[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:22,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:22,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,942 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:22,942 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,942 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:22,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:22,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:22,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:22,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:22,946 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:22,946 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:22,946 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:22,946 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:22,946 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:22,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:22,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:22,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:22,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:22,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:22,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:22,953 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:22,953 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:22,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:22,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:22,962 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=567, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/52ef6f51fea149df908d4e75136f0222 2023-04-07 09:54:22,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/60c940d897b643529e37ea2b0ac45baa 2023-04-07 09:54:23,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/cf5c15f7b65249259a015449ae0b397c 2023-04-07 09:54:23,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/52ef6f51fea149df908d4e75136f0222 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52ef6f51fea149df908d4e75136f0222 2023-04-07 09:54:23,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52ef6f51fea149df908d4e75136f0222, entries=200, sequenceid=567, filesize=32.3 K 2023-04-07 09:54:23,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/60c940d897b643529e37ea2b0ac45baa as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/60c940d897b643529e37ea2b0ac45baa 2023-04-07 09:54:23,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/60c940d897b643529e37ea2b0ac45baa, entries=150, sequenceid=567, filesize=11.9 K 2023-04-07 09:54:23,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/cf5c15f7b65249259a015449ae0b397c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cf5c15f7b65249259a015449ae0b397c 2023-04-07 09:54:23,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cf5c15f7b65249259a015449ae0b397c, entries=150, sequenceid=567, filesize=11.9 K 2023-04-07 09:54:23,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~506.91 KB/519072, currentSize=13.42 KB/13740 for b48a920edd75a7abe23cbbe5f3bc5475 in 545ms, sequenceid=567, compaction requested=false 2023-04-07 09:54:23,027 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:23,053 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:23,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:23,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:23,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:23,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:23,060 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:23,060 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:23,060 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:23,060 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:23,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:23,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:23,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:23,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:23,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:23,080 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:23,088 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:23,088 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:23,088 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-07 09:54:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:23,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:23,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:23,156 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e86a7820293a475b93e7a6d56ac486be to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e86a7820293a475b93e7a6d56ac486be 2023-04-07 09:54:23,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680861263108, max timestamp=1680861263174Num uniques 859; 2023-04-07 09:54:23,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-07 09:54:23,196 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=574, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/09ced868673d41cea8fed40319f374fc 2023-04-07 09:54:23,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861323197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861323208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861323208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861323209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861323209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,259 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/4221353efd7346d6a3f7f7f780ac488f 2023-04-07 09:54:23,312 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/015dce71e0284ec48cba68f1e67def14 2023-04-07 09:54:23,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861323310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861323316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861323316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861323316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861323317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,321 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/09ced868673d41cea8fed40319f374fc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/09ced868673d41cea8fed40319f374fc 2023-04-07 09:54:23,329 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/09ced868673d41cea8fed40319f374fc, entries=50, sequenceid=574, filesize=11.7 K 2023-04-07 09:54:23,330 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/4221353efd7346d6a3f7f7f780ac488f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4221353efd7346d6a3f7f7f780ac488f 2023-04-07 09:54:23,338 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4221353efd7346d6a3f7f7f780ac488f, entries=50, sequenceid=574, filesize=7.1 K 2023-04-07 09:54:23,339 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/015dce71e0284ec48cba68f1e67def14 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/015dce71e0284ec48cba68f1e67def14 2023-04-07 09:54:23,350 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/015dce71e0284ec48cba68f1e67def14, entries=50, sequenceid=574, filesize=7.1 K 2023-04-07 09:54:23,351 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=174.43 KB/178620 for b48a920edd75a7abe23cbbe5f3bc5475 in 263ms, sequenceid=574, compaction requested=true 2023-04-07 09:54:23,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:23,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:23,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:23,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:23,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:23,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:23,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:23,354 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,354 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:23,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:23,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d99fd35[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:23,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:23,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:23,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:23,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:23,364 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,364 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:23,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:23,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:23,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:23,367 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:23,372 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:23,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:23,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:23,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:23,372 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:23,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:23,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:23,381 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,382 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:23,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:23,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:23,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:23,386 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:23,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:23,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:23,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:23,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,387 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:23,387 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:23,487 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:23,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:23,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:23,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:23,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:23,490 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:23,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:23,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:23,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:23,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:23,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:23,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:23,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:23,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:23,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:23,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:23,500 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:23,500 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:23,500 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:23,500 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:23,500 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=174.43 KB heapSize=492.58 KB 2023-04-07 09:54:23,500 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:23,500 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:23,501 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:23,501 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:23,501 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:23,501 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:23,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:23,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:23,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861323518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861323520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861323520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861323520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861323521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861323621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861323822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861323822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861323823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861323824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:23,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861323826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:23,919 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407d5a1fe42dfd34e55b95eba6d177382a1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d5a1fe42dfd34e55b95eba6d177382a1 2023-04-07 09:54:24,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:24,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58782 deadline: 1680861324127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,324 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=603, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/aac46a5cd3074c87acc8af9e2342ccab 2023-04-07 09:54:24,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861324328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861324329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:24,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861324330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:24,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861324331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,337 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/0606e8d6aee345cdbfcc80d967bf7fad 2023-04-07 09:54:24,352 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a44ad1edc53a4fba95bb72fd155d82e2 2023-04-07 09:54:24,360 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/aac46a5cd3074c87acc8af9e2342ccab as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/aac46a5cd3074c87acc8af9e2342ccab 2023-04-07 09:54:24,365 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/aac46a5cd3074c87acc8af9e2342ccab, entries=150, sequenceid=603, filesize=25.4 K 2023-04-07 09:54:24,366 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/0606e8d6aee345cdbfcc80d967bf7fad as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0606e8d6aee345cdbfcc80d967bf7fad 2023-04-07 09:54:24,377 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0606e8d6aee345cdbfcc80d967bf7fad, entries=150, sequenceid=603, filesize=11.9 K 2023-04-07 09:54:24,378 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a44ad1edc53a4fba95bb72fd155d82e2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a44ad1edc53a4fba95bb72fd155d82e2 2023-04-07 09:54:24,382 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a44ad1edc53a4fba95bb72fd155d82e2, entries=150, sequenceid=603, filesize=11.9 K 2023-04-07 09:54:24,383 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~492.53 KB/504352, currentSize=6.71 KB/6870 for b48a920edd75a7abe23cbbe5f3bc5475 in 883ms, sequenceid=603, compaction requested=true 2023-04-07 09:54:24,383 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:24,383 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:24,385 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:24,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:24,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:24,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:24,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e71ff2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:24,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:24,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,388 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:24,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:24,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:24,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:24,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:24,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:24,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:24,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:24,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:24,393 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:24,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:24,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:24,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:24,393 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:24,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:24,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,398 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:24,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:24,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:24,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:24,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:24,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:24,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:24,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:24,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:24,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:24,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:24,403 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:24,403 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:24,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:24,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:24,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:24,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:24,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:24,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,503 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:24,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:24,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:24,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:24,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:24,507 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:24,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:24,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:24,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:24,508 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,508 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:24,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,509 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:24,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:24,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:24,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:24,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:24,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:24,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:24,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:24,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:24,524 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:24,524 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:24,525 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-07 09:54:24,525 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:24,525 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:24,525 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:24,525 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:24,525 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:24,525 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:24,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:24,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:24,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861264843, max timestamp=1680861264860Num uniques 859; 2023-04-07 09:54:24,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:24,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861264843, max timestamp=1680861264860Num uniques 859; 2023-04-07 09:54:24,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861264843, max timestamp=1680861264860Num uniques 859; 2023-04-07 09:54:24,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:24,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:24,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861264904, max timestamp=1680861264916Num uniques 1068; 2023-04-07 09:54:24,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:54:24,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861264904, max timestamp=1680861264916Num uniques 1068; 2023-04-07 09:54:24,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861264904, max timestamp=1680861264916Num uniques 1068; 2023-04-07 09:54:24,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:54:24,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:54:24,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861264947, max timestamp=1680861264985Num uniques 1027; 2023-04-07 09:54:24,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:24,988 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a65275748ad64bee83f132b0f33001ca to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a65275748ad64bee83f132b0f33001ca 2023-04-07 09:54:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861264947, max timestamp=1680861264985Num uniques 1027; 2023-04-07 09:54:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861264947, max timestamp=1680861264985Num uniques 1027; 2023-04-07 09:54:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:24,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:24,994 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=607, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/62ea0166b2e0432683094188ba3daea0 2023-04-07 09:54:25,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680861265020, max timestamp=1680861265036Num uniques 1086; 2023-04-07 09:54:25,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 09:54:25,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680861265020, max timestamp=1680861265036Num uniques 1086; 2023-04-07 09:54:25,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 09:54:25,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680861265020, max timestamp=1680861265036Num uniques 1086; 2023-04-07 09:54:25,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-07 09:54:25,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861265066, max timestamp=1680861265098Num uniques 1045; 2023-04-07 09:54:25,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-07 09:54:25,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861265066, max timestamp=1680861265098Num uniques 1045; 2023-04-07 09:54:25,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-07 09:54:25,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861265066, max timestamp=1680861265098Num uniques 1045; 2023-04-07 09:54:25,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-07 09:54:25,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680861265136, max timestamp=1680861265162Num uniques 1154; 2023-04-07 09:54:25,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-07 09:54:25,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861265150, max timestamp=1680861265162Num uniques 1154; 2023-04-07 09:54:25,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-07 09:54:25,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861265150, max timestamp=1680861265162Num uniques 1154; 2023-04-07 09:54:25,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-07 09:54:25,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680861265203, max timestamp=1680861265222Num uniques 1013; 2023-04-07 09:54:25,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 09:54:25,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680861265203, max timestamp=1680861265222Num uniques 1063; 2023-04-07 09:54:25,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-07 09:54:25,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680861265203, max timestamp=1680861265222Num uniques 1013; 2023-04-07 09:54:25,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-07 09:54:25,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680861265253, max timestamp=1680861265281Num uniques 1022; 2023-04-07 09:54:25,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-07 09:54:25,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680861265253, max timestamp=1680861265281Num uniques 1022; 2023-04-07 09:54:25,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-07 09:54:25,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680861265253, max timestamp=1680861265281Num uniques 1022; 2023-04-07 09:54:25,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-07 09:54:25,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861265315, max timestamp=1680861265333Num uniques 1081; 2023-04-07 09:54:25,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-07 09:54:25,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861265315, max timestamp=1680861265333Num uniques 1081; 2023-04-07 09:54:25,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-07 09:54:25,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861265315, max timestamp=1680861265333Num uniques 1081; 2023-04-07 09:54:25,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-07 09:54:25,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=440, cellSize=20150, totalHeapSize=46000, min timestamp=1680861265344, max timestamp=1680861265357Num uniques 1140; 2023-04-07 09:54:25,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=20150, suffix heap size=146528,new segment heap size=46000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=440 2023-04-07 09:54:25,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=440, cellSize=20150, totalHeapSize=46000, min timestamp=1680861265344, max timestamp=1680861265357Num uniques 1140; 2023-04-07 09:54:25,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=20150, suffix heap size=146528,new segment heap size=46000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=440 2023-04-07 09:54:25,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=440, cellSize=20150, totalHeapSize=46000, min timestamp=1680861265344, max timestamp=1680861265357Num uniques 1140; 2023-04-07 09:54:25,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=20150, suffix heap size=146528,new segment heap size=46000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=440 2023-04-07 09:54:25,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1680861265360, max timestamp=1680861265376Num uniques 1299; 2023-04-07 09:54:25,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=59494, new segment data size=15984, suffix heap size=163064,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1299, new segment cells count=349 2023-04-07 09:54:25,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1680861265360, max timestamp=1680861265376Num uniques 1299; 2023-04-07 09:54:25,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=59494, new segment data size=15984, suffix heap size=163064,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1299, new segment cells count=349 2023-04-07 09:54:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15984, totalHeapSize=36536, min timestamp=1680861265360, max timestamp=1680861265376Num uniques 1299; 2023-04-07 09:54:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=59494, new segment data size=15984, suffix heap size=163064,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1299, new segment cells count=349 2023-04-07 09:54:25,409 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/631bd18d77e8449486a3166aed010b2e 2023-04-07 09:54:25,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14098, totalHeapSize=32272, min timestamp=1680861265411, max timestamp=1680861265427Num uniques 1208; 2023-04-07 09:54:25,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55318, new segment data size=14098, suffix heap size=153600,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=308 2023-04-07 09:54:25,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14098, totalHeapSize=32272, min timestamp=1680861265411, max timestamp=1680861265427Num uniques 1208; 2023-04-07 09:54:25,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=55318, new segment data size=14098, suffix heap size=153600,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=308 2023-04-07 09:54:25,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861265420, max timestamp=1680861265427Num uniques 1208; 2023-04-07 09:54:25,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55318, new segment data size=6870, suffix heap size=153600,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=150 2023-04-07 09:54:25,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680861265438, max timestamp=1680861265452Num uniques 1167; 2023-04-07 09:54:25,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680861265438, max timestamp=1680861265452Num uniques 1009; 2023-04-07 09:54:25,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680861265438, max timestamp=1680861265452Num uniques 1167; 2023-04-07 09:54:25,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=16802, suffix heap size=132904,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=367 2023-04-07 09:54:25,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=16802, suffix heap size=149336,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=367 2023-04-07 09:54:25,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=16802, suffix heap size=149336,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=367 2023-04-07 09:54:25,465 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a9e3e4e5f6a344b89b380e62f3798572 2023-04-07 09:54:25,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/62ea0166b2e0432683094188ba3daea0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/62ea0166b2e0432683094188ba3daea0 2023-04-07 09:54:25,477 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/62ea0166b2e0432683094188ba3daea0, entries=50, sequenceid=607, filesize=11.7 K 2023-04-07 09:54:25,478 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/631bd18d77e8449486a3166aed010b2e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/631bd18d77e8449486a3166aed010b2e 2023-04-07 09:54:25,483 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/631bd18d77e8449486a3166aed010b2e, entries=50, sequenceid=607, filesize=7.1 K 2023-04-07 09:54:25,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680861265470, max timestamp=1680861265480Num uniques 1226; 2023-04-07 09:54:25,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=10346, suffix heap size=155472,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=226 2023-04-07 09:54:25,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680861265470, max timestamp=1680861265480Num uniques 1226; 2023-04-07 09:54:25,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=10346, suffix heap size=155472,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=226 2023-04-07 09:54:25,485 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a9e3e4e5f6a344b89b380e62f3798572 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a9e3e4e5f6a344b89b380e62f3798572 2023-04-07 09:54:25,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680861265470, max timestamp=1680861265480Num uniques 1226; 2023-04-07 09:54:25,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=10346, suffix heap size=155472,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=226 2023-04-07 09:54:25,491 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a9e3e4e5f6a344b89b380e62f3798572, entries=50, sequenceid=607, filesize=7.1 K 2023-04-07 09:54:25,494 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=87.22 KB/89310 for b48a920edd75a7abe23cbbe5f3bc5475 in 969ms, sequenceid=607, compaction requested=true 2023-04-07 09:54:25,494 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:25,494 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:25,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:25,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=93.93 KB heapSize=259.17 KB 2023-04-07 09:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e6a5f6f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:25,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:25,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:25,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:25,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:25,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:25,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:25,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:25,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:25,506 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,506 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:25,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,508 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:25,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:25,511 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:25,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:25,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:25,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:25,511 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:25,514 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,514 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,514 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:25,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:25,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:25,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,521 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,522 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:25,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:25,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,522 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:25,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861325513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861325513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861325520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861325521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861325522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,622 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:25,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:25,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:25,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:25,626 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:25,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:25,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:25,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:25,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861325623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861325623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861325624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,628 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:25,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:25,628 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:25,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:25,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:25,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:25,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:25,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:25,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:25,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861325629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861325629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:25,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:25,638 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:25,638 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:25,639 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:25,639 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:25,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:25,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:25,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:25,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:25,640 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,640 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,641 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,641 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20847a8d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:25,642 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:25,642 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:25,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:25,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:25,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:25,648 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:25,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:25,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:25,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:25,648 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:25,650 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,650 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,650 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,651 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:25,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:25,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:25,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:25,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:25,655 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:25,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,655 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:25,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,755 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:25,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:25,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:25,758 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:25,758 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:25,758 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:25,758 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:25,758 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,758 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,760 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,760 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:25,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:25,760 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,760 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:25,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:25,761 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:25,762 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:25,762 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:25,762 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:25,762 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:25,764 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:25,766 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:25,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32664bfd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:25,766 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,767 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,767 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:25,767 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:25,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:25,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:25,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:25,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:25,771 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:25,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:25,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:25,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:25,771 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:25,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,773 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:25,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:25,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:25,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:25,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,777 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:25,777 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,777 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:25,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:25,777 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861325830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861325830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58752 deadline: 1680861325830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861325834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861325834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,877 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:25,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:25,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:25,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:25,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:25,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:25,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,885 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:25,885 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:25,885 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,885 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:25,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:25,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:25,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:25,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:25,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:25,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:25,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:25,886 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:25,887 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:25,887 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:25,887 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,889 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,889 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:25,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:25,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe3dacc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:25,891 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:25,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:25,896 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:25,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:25,899 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:25,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:25,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:25,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:25,899 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:25,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:25,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:25,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:25,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:25,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:25,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:25,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:25,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:25,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:25,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:25,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:25,905 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:25,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:25,905 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:25,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:25,934 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407fe213b99e9b24e6b9b2c140f500b72f6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fe213b99e9b24e6b9b2c140f500b72f6 2023-04-07 09:54:26,005 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,009 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,010 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:26,012 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:26,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,015 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:26,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:26,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23fe7978[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:26,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:26,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:26,019 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:26,022 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:26,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:26,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:26,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:26,022 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:26,024 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,024 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,024 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:26,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:26,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,028 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:26,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:26,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,029 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:26,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,129 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,132 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,132 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,132 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,132 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,133 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,133 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861326134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861326134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861326135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861326138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,139 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,139 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,140 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:26,140 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:26,141 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861326139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,141 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,142 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,142 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202d3f98[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:26,143 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:26,143 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,144 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:26,147 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,147 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,147 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,147 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:26,150 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:26,150 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:26,150 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:26,150 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:26,150 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:26,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:26,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:26,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,156 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,156 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,156 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,156 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,155 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:26,156 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,156 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:26,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,256 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,259 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,260 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,260 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,261 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,261 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,261 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,261 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,263 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,263 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,263 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,263 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,263 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,263 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,263 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,264 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:26,264 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:26,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:26,266 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,266 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,266 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,266 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2014b1c3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:26,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:26,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,269 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:26,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:26,270 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:26,273 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:26,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:26,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:26,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:26,273 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:26,276 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,276 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,276 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,281 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:26,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,286 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:26,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,286 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:26,349 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=861, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/ad6036e2f4f44bd2ad21d8ed99839061 2023-04-07 09:54:26,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/7d1696648791425bbcb618850a234d32 2023-04-07 09:54:26,387 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,400 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,409 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,409 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,409 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,416 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,416 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,416 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:26,416 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:26,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,418 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:26,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11902ab7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:26,421 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:26,421 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,422 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:26,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,423 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:26,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:26,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:26,427 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:26,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:26,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:26,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:26,427 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:26,430 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,430 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,430 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,431 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,433 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:26,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:26,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,436 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:26,436 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:26,436 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,436 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,536 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,540 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,542 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,544 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,544 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,544 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:26,544 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,546 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:26,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4175a444[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:26,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:26,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:26,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:26,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:26,550 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,550 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:26,551 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:26,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:26,554 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:26,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:26,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:26,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:26,554 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:26,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:26,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:26,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,565 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:26,566 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:26,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,565 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861326637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861326639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861326639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861326642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861326644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,666 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,670 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,673 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,673 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,673 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:26,674 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:26,674 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:26,675 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:26,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86d920d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:26,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:26,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,679 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:26,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:26,679 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:26,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:26,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:26,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:26,685 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:26,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:26,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:26,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:26,685 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:26,686 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,686 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,686 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,690 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:26,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,695 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:26,695 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:26,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,795 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,799 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,805 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,805 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,805 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:26,805 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,806 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:26,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,810 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,810 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/e479b069286940eca05e9f28702353c2 2023-04-07 09:54:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:26,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@df2a06a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:26,812 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:26,812 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,814 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:26,814 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,814 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,815 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:26,815 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/ad6036e2f4f44bd2ad21d8ed99839061 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad6036e2f4f44bd2ad21d8ed99839061 2023-04-07 09:54:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:26,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:26,820 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:26,820 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:26,820 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:26,820 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:26,820 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:26,823 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,823 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,823 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:26,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad6036e2f4f44bd2ad21d8ed99839061, entries=250, sequenceid=861, filesize=39.2 K 2023-04-07 09:54:26,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/7d1696648791425bbcb618850a234d32 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7d1696648791425bbcb618850a234d32 2023-04-07 09:54:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,827 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:26,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:26,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:26,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,830 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:26,830 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:26,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:26,831 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7d1696648791425bbcb618850a234d32, entries=150, sequenceid=861, filesize=11.9 K 2023-04-07 09:54:26,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/e479b069286940eca05e9f28702353c2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/e479b069286940eca05e9f28702353c2 2023-04-07 09:54:26,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/e479b069286940eca05e9f28702353c2, entries=150, sequenceid=861, filesize=11.9 K 2023-04-07 09:54:26,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~279.05 KB/285744, currentSize=87.22 KB/89310 for b48a920edd75a7abe23cbbe5f3bc5475 in 1341ms, sequenceid=861, compaction requested=true 2023-04-07 09:54:26,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:26,837 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:26,837 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 09:54:26,837 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 09:54:26,837 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:26,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:26,840 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 149514 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 09:54:26,841 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:26,841 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,841 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/c6512d87ef104798b247e7c69f96e5c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52ef6f51fea149df908d4e75136f0222, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/09ced868673d41cea8fed40319f374fc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/aac46a5cd3074c87acc8af9e2342ccab, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/62ea0166b2e0432683094188ba3daea0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad6036e2f4f44bd2ad21d8ed99839061] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=146.0 K 2023-04-07 09:54:26,841 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting c6512d87ef104798b247e7c69f96e5c1, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680861261335 2023-04-07 09:54:26,842 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 63638 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 09:54:26,842 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 52ef6f51fea149df908d4e75136f0222, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680861262363 2023-04-07 09:54:26,842 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:26,842 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,842 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/658ac96170734bbeaab79d378ec2b93a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/60c940d897b643529e37ea2b0ac45baa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4221353efd7346d6a3f7f7f780ac488f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0606e8d6aee345cdbfcc80d967bf7fad, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/631bd18d77e8449486a3166aed010b2e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7d1696648791425bbcb618850a234d32] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=62.1 K 2023-04-07 09:54:26,843 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 09ced868673d41cea8fed40319f374fc, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680861262485 2023-04-07 09:54:26,843 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 658ac96170734bbeaab79d378ec2b93a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680861261335 2023-04-07 09:54:26,843 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting aac46a5cd3074c87acc8af9e2342ccab, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1680861263179 2023-04-07 09:54:26,843 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 60c940d897b643529e37ea2b0ac45baa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680861262363 2023-04-07 09:54:26,846 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4221353efd7346d6a3f7f7f780ac488f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680861262485 2023-04-07 09:54:26,847 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 62ea0166b2e0432683094188ba3daea0, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1680861263517 2023-04-07 09:54:26,847 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0606e8d6aee345cdbfcc80d967bf7fad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1680861263179 2023-04-07 09:54:26,847 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ad6036e2f4f44bd2ad21d8ed99839061, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1680861265486 2023-04-07 09:54:26,847 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 631bd18d77e8449486a3166aed010b2e, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1680861263517 2023-04-07 09:54:26,848 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7d1696648791425bbcb618850a234d32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1680861265486 2023-04-07 09:54:26,870 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#196 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:26,884 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#195 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:26,931 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:26,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:26,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:26,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:26,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:26,944 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:26,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:26,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:26,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:26,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:26,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:26,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:26,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:26,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:26,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:26,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:26,956 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:26,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:26,958 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:26,958 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:27,290 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/bfd0c696ac614ecd8857e67e97954bdd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bfd0c696ac614ecd8857e67e97954bdd 2023-04-07 09:54:27,296 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into bfd0c696ac614ecd8857e67e97954bdd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:27,296 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:27,296 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=10, startTime=1680861266837; duration=0sec 2023-04-07 09:54:27,296 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:27,296 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-07 09:54:27,298 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 63638 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-07 09:54:27,298 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:27,298 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:27,298 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/31b552e992c04a1e801d8cd65e5987cf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cf5c15f7b65249259a015449ae0b397c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/015dce71e0284ec48cba68f1e67def14, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a44ad1edc53a4fba95bb72fd155d82e2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a9e3e4e5f6a344b89b380e62f3798572, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/e479b069286940eca05e9f28702353c2] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=62.1 K 2023-04-07 09:54:27,299 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 31b552e992c04a1e801d8cd65e5987cf, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680861261335 2023-04-07 09:54:27,299 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cf5c15f7b65249259a015449ae0b397c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680861262363 2023-04-07 09:54:27,299 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 015dce71e0284ec48cba68f1e67def14, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680861262485 2023-04-07 09:54:27,300 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a44ad1edc53a4fba95bb72fd155d82e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1680861263179 2023-04-07 09:54:27,300 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a9e3e4e5f6a344b89b380e62f3798572, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1680861263517 2023-04-07 09:54:27,300 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e479b069286940eca05e9f28702353c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1680861265486 2023-04-07 09:54:27,311 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:27,327 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/2147c77c4c76453e98c7c10083daafef as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/2147c77c4c76453e98c7c10083daafef 2023-04-07 09:54:27,332 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into 2147c77c4c76453e98c7c10083daafef(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:27,332 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:27,332 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=10, startTime=1680861266838; duration=0sec 2023-04-07 09:54:27,332 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:27,401 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304072f1e63131a85460f8a0dad0c2baef31a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072f1e63131a85460f8a0dad0c2baef31a 2023-04-07 09:54:27,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:27,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:27,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861327654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861327655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861327655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861327656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861327657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58774 deadline: 1680861327758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861327758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861327761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861327761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861327762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,809 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=877, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/554c1c56b9e54756b19c83c7133a415c 2023-04-07 09:54:27,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861327961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861327961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861327965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861327966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:27,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:27,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861327967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,106 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/4983f7046a5a40e699d182eb3cf1410e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4983f7046a5a40e699d182eb3cf1410e 2023-04-07 09:54:28,112 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into 4983f7046a5a40e699d182eb3cf1410e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-07 09:54:28,112 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:28,112 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=10, startTime=1680861266837; duration=1sec 2023-04-07 09:54:28,112 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:28,219 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/311549c965b849f69404cfdc2bc955f7 2023-04-07 09:54:28,230 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/94a87531ce6d4cc498347e69e7315187 2023-04-07 09:54:28,234 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/554c1c56b9e54756b19c83c7133a415c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/554c1c56b9e54756b19c83c7133a415c 2023-04-07 09:54:28,237 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/554c1c56b9e54756b19c83c7133a415c, entries=150, sequenceid=877, filesize=25.4 K 2023-04-07 09:54:28,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/311549c965b849f69404cfdc2bc955f7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/311549c965b849f69404cfdc2bc955f7 2023-04-07 09:54:28,242 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/311549c965b849f69404cfdc2bc955f7, entries=150, sequenceid=877, filesize=11.9 K 2023-04-07 09:54:28,243 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/94a87531ce6d4cc498347e69e7315187 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/94a87531ce6d4cc498347e69e7315187 2023-04-07 09:54:28,247 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/94a87531ce6d4cc498347e69e7315187, entries=150, sequenceid=877, filesize=11.9 K 2023-04-07 09:54:28,248 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b48a920edd75a7abe23cbbe5f3bc5475 in 1289ms, sequenceid=877, compaction requested=false 2023-04-07 09:54:28,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:28,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:28,249 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,249 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:28,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da7f6f1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:28,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:28,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:28,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:28,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:28,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:28,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:28,259 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:28,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:28,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:28,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:28,259 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:28,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,261 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,262 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,264 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,264 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,264 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,264 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,264 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,264 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,264 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:28,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:28,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:28,265 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:28,265 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:28,265 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,265 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:28,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-07 09:54:28,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:28,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:28,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:28,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:28,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:28,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:28,287 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040759dfabb32aa240f1aed7d44231ca428f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040759dfabb32aa240f1aed7d44231ca428f 2023-04-07 09:54:28,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861328288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861328290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861328290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861328290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861328291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,365 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:28,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:28,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:28,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:28,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:28,369 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:28,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:28,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:28,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:28,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:28,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:28,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:28,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:28,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:28,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:28,373 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,373 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:28,373 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:28,373 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:28,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:28,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:28,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:28,374 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,374 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:28,376 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:28,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@719c3685[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:28,376 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,377 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:28,377 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:28,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:28,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:28,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:28,382 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:28,382 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:28,382 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:28,382 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:28,382 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:28,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,385 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:28,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,390 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,390 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:28,390 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:28,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861328396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58814 deadline: 1680861328396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861328397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861328397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,490 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:28,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:28,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:28,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:28,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:28,494 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:28,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:28,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:28,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,499 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,499 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:28,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,500 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:28,504 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,504 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:28,504 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:28,504 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:28,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:28,507 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,507 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,507 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,508 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:28,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15052f41[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:28,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:28,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,510 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:28,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:28,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:28,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:28,511 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,511 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:28,511 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,511 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:28,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:28,514 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:28,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:28,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:28,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:28,514 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:28,515 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,515 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,515 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,516 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,520 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,520 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,520 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:28,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:28,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:28,520 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,520 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,521 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:28,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,521 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:28,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861328598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861328599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861328600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861328600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,621 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:28,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:28,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:28,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:28,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:28,624 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,626 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:28,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:28,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:28,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:28,628 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,628 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:28,628 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:28,628 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:28,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:28,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,629 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:28,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d6c3c7b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:28,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,632 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:28,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:28,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:28,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:28,638 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:28,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:28,640 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:28,641 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:28,641 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:28,641 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:28,641 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:28,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:28,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,642 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:28,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,647 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:28,647 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:28,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:28,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:28,647 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:28,694 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=898, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/7201032c974f434296ec3f3375d1f3c7 2023-04-07 09:54:28,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=898 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/db010813b7c84a00a6adbf25bf0af5ac 2023-04-07 09:54:28,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=898 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a16e11998258484eb7fdd7da38533e42 2023-04-07 09:54:28,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/7201032c974f434296ec3f3375d1f3c7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/7201032c974f434296ec3f3375d1f3c7 2023-04-07 09:54:28,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/7201032c974f434296ec3f3375d1f3c7, entries=200, sequenceid=898, filesize=32.3 K 2023-04-07 09:54:28,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/db010813b7c84a00a6adbf25bf0af5ac as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/db010813b7c84a00a6adbf25bf0af5ac 2023-04-07 09:54:28,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/db010813b7c84a00a6adbf25bf0af5ac, entries=150, sequenceid=898, filesize=11.9 K 2023-04-07 09:54:28,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a16e11998258484eb7fdd7da38533e42 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a16e11998258484eb7fdd7da38533e42 2023-04-07 09:54:28,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a16e11998258484eb7fdd7da38533e42, entries=150, sequenceid=898, filesize=11.9 K 2023-04-07 09:54:28,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for b48a920edd75a7abe23cbbe5f3bc5475 in 477ms, sequenceid=898, compaction requested=true 2023-04-07 09:54:28,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:28,746 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:28,747 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:28,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:28,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:28,747 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:28,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:28,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:28,748 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:28,749 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:28,749 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:28,749 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,749 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:28,749 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4983f7046a5a40e699d182eb3cf1410e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/554c1c56b9e54756b19c83c7133a415c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/7201032c974f434296ec3f3375d1f3c7] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=83.8 K 2023-04-07 09:54:28,749 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,749 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bfd0c696ac614ecd8857e67e97954bdd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/311549c965b849f69404cfdc2bc955f7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/db010813b7c84a00a6adbf25bf0af5ac] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=36.3 K 2023-04-07 09:54:28,749 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4983f7046a5a40e699d182eb3cf1410e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1680861265486 2023-04-07 09:54:28,749 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting bfd0c696ac614ecd8857e67e97954bdd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1680861265486 2023-04-07 09:54:28,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:28,750 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 311549c965b849f69404cfdc2bc955f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1680861265510 2023-04-07 09:54:28,750 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 554c1c56b9e54756b19c83c7133a415c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1680861265510 2023-04-07 09:54:28,750 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting db010813b7c84a00a6adbf25bf0af5ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1680861267653 2023-04-07 09:54:28,750 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 7201032c974f434296ec3f3375d1f3c7, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1680861267653 2023-04-07 09:54:28,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:28,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:28,751 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:28,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:28,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:28,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,753 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:28,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:28,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:28,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:28,755 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:28,755 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:28,755 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:28,760 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,760 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:28,764 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,764 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:28,764 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-07 09:54:28,765 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:28,765 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:28,765 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:28,765 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:28,765 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:28,765 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:28,773 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#204 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:28,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:28,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:28,825 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#205 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:28,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861328832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,856 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/a27abd485b4847559a94a81bbfd1233b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a27abd485b4847559a94a81bbfd1233b 2023-04-07 09:54:28,859 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040757337cb643b44c4d90df52c53b66a173 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040757337cb643b44c4d90df52c53b66a173 2023-04-07 09:54:28,867 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into a27abd485b4847559a94a81bbfd1233b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:28,867 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:28,867 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=13, startTime=1680861268747; duration=0sec 2023-04-07 09:54:28,867 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:28,867 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:28,868 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:28,868 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:28,869 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:28,869 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/2147c77c4c76453e98c7c10083daafef, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/94a87531ce6d4cc498347e69e7315187, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a16e11998258484eb7fdd7da38533e42] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=36.3 K 2023-04-07 09:54:28,869 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2147c77c4c76453e98c7c10083daafef, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1680861265486 2023-04-07 09:54:28,869 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 94a87531ce6d4cc498347e69e7315187, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1680861265510 2023-04-07 09:54:28,872 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a16e11998258484eb7fdd7da38533e42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1680861267653 2023-04-07 09:54:28,879 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=913, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/d0550812d5964d15a335d6c3ed48a2d8 2023-04-07 09:54:28,881 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:28,884 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/1c7a61d543374220afd32c474527afd9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1c7a61d543374220afd32c474527afd9 2023-04-07 09:54:28,891 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into 1c7a61d543374220afd32c474527afd9(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:28,891 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:28,891 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=13, startTime=1680861268746; duration=0sec 2023-04-07 09:54:28,891 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:28,894 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=913 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/6fe5737d57ae4228b45a84fba259ee1b 2023-04-07 09:54:28,895 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/1f844a8663b546f9a39eee7f13297a8a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/1f844a8663b546f9a39eee7f13297a8a 2023-04-07 09:54:28,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861328902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861328908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861328908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,909 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into 1f844a8663b546f9a39eee7f13297a8a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:28,909 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:28,909 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=13, startTime=1680861268747; duration=0sec 2023-04-07 09:54:28,910 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:28,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861328909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:28,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861328939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861329141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,322 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=913 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/006b96a2c7134de0a86fb3f706518e1b 2023-04-07 09:54:29,327 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/d0550812d5964d15a335d6c3ed48a2d8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d0550812d5964d15a335d6c3ed48a2d8 2023-04-07 09:54:29,330 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d0550812d5964d15a335d6c3ed48a2d8, entries=150, sequenceid=913, filesize=25.4 K 2023-04-07 09:54:29,331 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/6fe5737d57ae4228b45a84fba259ee1b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/6fe5737d57ae4228b45a84fba259ee1b 2023-04-07 09:54:29,334 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/6fe5737d57ae4228b45a84fba259ee1b, entries=150, sequenceid=913, filesize=11.9 K 2023-04-07 09:54:29,335 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/006b96a2c7134de0a86fb3f706518e1b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/006b96a2c7134de0a86fb3f706518e1b 2023-04-07 09:54:29,339 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/006b96a2c7134de0a86fb3f706518e1b, entries=150, sequenceid=913, filesize=11.9 K 2023-04-07 09:54:29,340 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for b48a920edd75a7abe23cbbe5f3bc5475 in 576ms, sequenceid=913, compaction requested=false 2023-04-07 09:54:29,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:29,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,342 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fde5359[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:29,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:29,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,345 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:29,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,347 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:29,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:29,349 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:29,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:29,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:29,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:29,350 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:29,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,353 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:29,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,356 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:29,356 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:29,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:29,410 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 09:54:29,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:29,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:29,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:29,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:29,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:29,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:29,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861329420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861329425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861329426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861329426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861329445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,456 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:29,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:29,460 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:29,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:29,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:29,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,461 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:29,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:29,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:29,464 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,464 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:29,464 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:29,464 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:29,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:29,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe67039[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:29,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:29,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,478 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:29,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:29,479 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:29,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:29,481 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:29,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:29,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:29,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:29,482 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:29,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:29,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:29,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,488 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:29,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,488 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:29,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861329526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861329528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861329528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861329528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,588 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:29,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:29,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:29,592 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:29,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:29,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:29,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:29,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:29,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:29,596 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,596 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:29,596 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:29,596 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:29,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:29,598 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,598 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,598 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:29,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3253fa3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:29,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:29,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,600 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:29,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:29,601 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,601 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:29,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:29,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:29,605 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:29,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:29,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:29,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:29,605 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:29,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,608 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:29,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:29,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,612 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:29,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,612 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:29,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,712 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:29,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:29,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:29,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:29,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:29,718 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:29,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:29,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:29,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:29,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:29,721 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:29,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:29,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:29,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:29,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,724 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:29,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fc65458[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:29,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:29,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,730 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,730 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:29,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861329732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861329732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861329732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:29,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:29,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:29,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:29,737 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861329736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:29,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:29,742 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:29,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:29,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:29,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:29,742 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:29,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,744 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,748 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:29,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,748 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:29,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,748 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:29,826 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040727bd71bdd3d640949af1a1e4627cb126 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040727bd71bdd3d640949af1a1e4627cb126 2023-04-07 09:54:29,848 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:29,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:29,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:29,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:29,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:29,852 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:29,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:29,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:29,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:29,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:29,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:29,857 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,857 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:29,857 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:29,857 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:29,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:29,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:29,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32df8662[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:29,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:29,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:29,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:29,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:29,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:29,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,866 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:29,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:29,869 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:29,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:29,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:29,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:29,869 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:29,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,873 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,875 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:29,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:29,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:29,875 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:29,875 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:29,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:29,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 483 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861329946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,975 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:29,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:29,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:29,978 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:29,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:29,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:29,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,980 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:29,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:29,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:29,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:29,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:29,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:29,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:29,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:29,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:29,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:29,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:29,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:29,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:29,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:29,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:29,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:29,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:29,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:29,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:29,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:29,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e345fb8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:29,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,996 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,996 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:29,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:29,997 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:29,997 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:29,997 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:29,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:30,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,002 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,002 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,002 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,002 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,002 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,011 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,011 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861330034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861330034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861330035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861330043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,112 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,115 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:30,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:30,118 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,120 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:30,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:30,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c6b7fa[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:30,122 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:30,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:30,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:30,122 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,127 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,127 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,128 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,128 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,128 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,129 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,132 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,132 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,132 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,232 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,234 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=934, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/52f51b6d02244d56bd5ac989cb7c3890 2023-04-07 09:54:30,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,235 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,237 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:30,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:30,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:30,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:30,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:30,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:30,243 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:30,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49ff55ae[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:30,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:30,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:30,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:30,252 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,253 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,253 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,253 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,253 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,256 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,263 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,263 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,263 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,364 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,367 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,369 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:30,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:30,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:30,370 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,370 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:30,371 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:30,371 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:30,372 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:30,374 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:30,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dd3665[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:30,374 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:30,375 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:30,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:30,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,378 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:30,378 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,378 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:30,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,380 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,380 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,380 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,380 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,380 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,384 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,386 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,386 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,386 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,486 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,490 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,491 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:30,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:30,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:30,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:30,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:30,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:30,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:30,494 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:30,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:30,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:30,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:30,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,496 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,497 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,497 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:30,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b0860b9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:30,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:30,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,499 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,499 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:30,499 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:30,499 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,499 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:30,499 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:30,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,503 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,503 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,503 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,503 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,503 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,505 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,518 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,522 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,522 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861330537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861330539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861330541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861330548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,622 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,626 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,627 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,627 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,628 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,628 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:30,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:30,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:30,629 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:30,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,630 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:30,630 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:30,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:30,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ad07d45[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:30,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:30,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,633 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:30,634 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,634 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:30,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,637 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:30,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,639 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,639 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,639 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,639 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,639 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,642 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,647 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,648 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,648 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=934 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/512a6e93c913440c953c3a9ebad7cb75 2023-04-07 09:54:30,749 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,752 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,755 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,755 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:30,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:30,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:30,756 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,756 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:30,756 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:30,756 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:30,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:30,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,757 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:30,759 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:30,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b11d94[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:30,759 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,760 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:30,760 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:30,760 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,761 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:30,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:30,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,762 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:30,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,764 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,764 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,767 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:30,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,770 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,771 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,771 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,771 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,871 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,874 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,875 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:30,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:30,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:30,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:30,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:30,878 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:30,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,879 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,879 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,879 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:30,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:30,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67317cd9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:30,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,883 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:30,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:30,885 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:30,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:30,887 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:30,887 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:30,887 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:30,887 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:30,887 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:30,889 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,889 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,889 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:30,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:30,889 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:30,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:30,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:30,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:30,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,893 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:30,893 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:30,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:30,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861330957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,993 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:30,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:30,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:30,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:30,997 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:30,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:30,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:30,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:30,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:30,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:30,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:30,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:30,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:30,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:31,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:31,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:31,000 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:31,001 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:31,001 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:31,001 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:31,001 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:31,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:31,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,002 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:31,003 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,003 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f64d52e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:31,004 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:31,004 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,006 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:31,006 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:31,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:31,007 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,007 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:31,007 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:31,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:31,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:31,010 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:31,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:31,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:31,010 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:31,010 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:31,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:31,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:31,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:31,013 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:31,014 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,016 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,016 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:31,016 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:31,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:31,016 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:31,016 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:31,017 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:31,017 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:31,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:31,017 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:31,017 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:31,017 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:31,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=934 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/71967e1334c5466ea71b9b27d9274753 2023-04-07 09:54:31,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/52f51b6d02244d56bd5ac989cb7c3890 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52f51b6d02244d56bd5ac989cb7c3890 2023-04-07 09:54:31,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52f51b6d02244d56bd5ac989cb7c3890, entries=200, sequenceid=934, filesize=32.3 K 2023-04-07 09:54:31,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/512a6e93c913440c953c3a9ebad7cb75 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/512a6e93c913440c953c3a9ebad7cb75 2023-04-07 09:54:31,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/512a6e93c913440c953c3a9ebad7cb75, entries=150, sequenceid=934, filesize=11.9 K 2023-04-07 09:54:31,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/71967e1334c5466ea71b9b27d9274753 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/71967e1334c5466ea71b9b27d9274753 2023-04-07 09:54:31,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/71967e1334c5466ea71b9b27d9274753, entries=150, sequenceid=934, filesize=11.9 K 2023-04-07 09:54:31,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for b48a920edd75a7abe23cbbe5f3bc5475 in 1670ms, sequenceid=934, compaction requested=true 2023-04-07 09:54:31,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:31,080 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:31,080 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:31,080 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:31,080 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:31,080 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:31,081 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85928 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:31,081 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:31,081 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:31,081 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:31,081 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:31,082 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1c7a61d543374220afd32c474527afd9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d0550812d5964d15a335d6c3ed48a2d8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52f51b6d02244d56bd5ac989cb7c3890] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=83.9 K 2023-04-07 09:54:31,082 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:31,082 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a27abd485b4847559a94a81bbfd1233b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/6fe5737d57ae4228b45a84fba259ee1b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/512a6e93c913440c953c3a9ebad7cb75] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=36.4 K 2023-04-07 09:54:31,082 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1c7a61d543374220afd32c474527afd9, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1680861267653 2023-04-07 09:54:31,082 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a27abd485b4847559a94a81bbfd1233b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1680861267653 2023-04-07 09:54:31,083 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d0550812d5964d15a335d6c3ed48a2d8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1680861268287 2023-04-07 09:54:31,083 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 6fe5737d57ae4228b45a84fba259ee1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1680861268287 2023-04-07 09:54:31,083 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 52f51b6d02244d56bd5ac989cb7c3890, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1680861268824 2023-04-07 09:54:31,083 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 512a6e93c913440c953c3a9ebad7cb75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1680861268824 2023-04-07 09:54:31,091 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:31,106 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#213 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:31,118 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:31,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:31,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:31,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:31,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:31,120 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:31,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:31,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:31,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:31,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:31,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:31,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:31,125 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,125 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:31,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:31,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:31,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:31,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:31,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:31,126 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:31,126 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:31,126 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-07 09:54:31,126 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:31,127 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:31,127 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:31,127 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:31,127 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:31,127 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:31,512 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/5982b5636bdc4278976765a0a40c7b9c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/5982b5636bdc4278976765a0a40c7b9c 2023-04-07 09:54:31,519 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into 5982b5636bdc4278976765a0a40c7b9c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:31,519 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:31,519 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=13, startTime=1680861271080; duration=0sec 2023-04-07 09:54:31,519 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:31,519 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:31,521 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:31,521 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:31,521 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:31,521 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/1f844a8663b546f9a39eee7f13297a8a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/006b96a2c7134de0a86fb3f706518e1b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/71967e1334c5466ea71b9b27d9274753] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=36.4 K 2023-04-07 09:54:31,522 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1f844a8663b546f9a39eee7f13297a8a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=898, earliestPutTs=1680861267653 2023-04-07 09:54:31,523 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 006b96a2c7134de0a86fb3f706518e1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1680861268287 2023-04-07 09:54:31,523 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 71967e1334c5466ea71b9b27d9274753, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1680861268824 2023-04-07 09:54:31,527 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/238715ef1b884057bc5a1d8fb6b088e4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/238715ef1b884057bc5a1d8fb6b088e4 2023-04-07 09:54:31,540 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into 238715ef1b884057bc5a1d8fb6b088e4(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:31,540 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:31,540 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=13, startTime=1680861271080; duration=0sec 2023-04-07 09:54:31,540 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:31,541 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#216 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:31,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:31,553 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304077ed04ba57e894dc9a2b017c8ae13ddfe to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077ed04ba57e894dc9a2b017c8ae13ddfe 2023-04-07 09:54:31,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861331567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861331569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861331570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861331571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,590 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=949, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/98d18c56fc7243a0a96e72888e9a9e4c 2023-04-07 09:54:31,604 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=949 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/f91ebc597eb64b06a65352486cb36f7b 2023-04-07 09:54:31,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861331672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861331672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861331674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861331674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861331874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861331876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861331878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:31,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861331879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:31,987 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/cc835d58f0b641bc8853b4472a6a64e2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cc835d58f0b641bc8853b4472a6a64e2 2023-04-07 09:54:31,991 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into cc835d58f0b641bc8853b4472a6a64e2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:31,991 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:31,991 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=13, startTime=1680861271080; duration=0sec 2023-04-07 09:54:31,992 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:32,015 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=949 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/7e0fdd3a9c3d4257a52c55fe1bd50260 2023-04-07 09:54:32,020 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/98d18c56fc7243a0a96e72888e9a9e4c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/98d18c56fc7243a0a96e72888e9a9e4c 2023-04-07 09:54:32,025 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/98d18c56fc7243a0a96e72888e9a9e4c, entries=150, sequenceid=949, filesize=25.4 K 2023-04-07 09:54:32,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/f91ebc597eb64b06a65352486cb36f7b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f91ebc597eb64b06a65352486cb36f7b 2023-04-07 09:54:32,037 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f91ebc597eb64b06a65352486cb36f7b, entries=150, sequenceid=949, filesize=11.9 K 2023-04-07 09:54:32,040 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/7e0fdd3a9c3d4257a52c55fe1bd50260 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7e0fdd3a9c3d4257a52c55fe1bd50260 2023-04-07 09:54:32,050 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7e0fdd3a9c3d4257a52c55fe1bd50260, entries=150, sequenceid=949, filesize=11.9 K 2023-04-07 09:54:32,051 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for b48a920edd75a7abe23cbbe5f3bc5475 in 925ms, sequenceid=949, compaction requested=false 2023-04-07 09:54:32,051 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:32,051 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,053 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@667bdd2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,057 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,057 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,057 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:32,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,066 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,066 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,067 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,067 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,067 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,069 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,070 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,073 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,073 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,073 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:32,074 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,074 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,074 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,173 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:32,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:32,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:32,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:32,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:32,178 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:32,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:32,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:32,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,180 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,181 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,181 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:32,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:32,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:32,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:32,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:32,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:32,183 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:32,183 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,183 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:32,183 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:32,183 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:32,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:32,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:32,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:32,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:32,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:32,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,188 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,188 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73b5d37[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,189 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,190 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,190 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,190 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,191 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:32,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,194 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,194 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,194 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,194 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,194 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,196 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,196 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,196 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861332195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,197 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861332195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,198 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861332197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861332197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:32,202 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,203 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,203 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861332298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861332298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861332300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861332300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,304 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:32,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:32,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:32,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:32,308 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:32,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:32,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:32,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:32,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:32,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:32,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:32,313 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,313 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:32,320 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:32,320 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,320 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,322 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,322 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,322 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b003ebd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,324 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,324 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,324 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,324 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,327 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:32,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,330 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,330 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,330 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,330 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,330 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,336 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,336 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,436 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:32,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:32,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:32,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:32,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:32,439 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:32,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:32,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:32,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,441 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:32,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:32,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,442 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:32,443 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,443 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:32,443 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:32,443 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72d44b56[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,448 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,448 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,461 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,461 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,463 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,463 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,463 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,468 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,468 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,468 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:32,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861332502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861332503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861332503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861332504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,568 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:32,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:32,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:32,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:32,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:32,579 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:32,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:32,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:32,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,581 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,581 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:32,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:32,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4966f1f3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,586 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,589 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,591 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,591 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,594 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,598 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,598 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,605 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a3cb3e711d36446897c671282f7efa4f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a3cb3e711d36446897c671282f7efa4f 2023-04-07 09:54:32,698 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:32,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:32,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:32,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:32,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:32,701 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:32,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:32,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:32,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,703 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:32,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:32,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:32,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:32,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:32,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:32,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:32,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:32,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:32,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1469df0d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,711 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,715 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,716 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,722 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,723 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,723 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861332807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861332807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861332807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861332807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,823 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:32,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:32,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:32,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:32,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:32,828 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:32,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:32,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:32,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:32,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,831 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:32,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:32,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:32,841 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,846 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e76feeb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,856 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,857 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,857 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,866 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,866 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,866 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,872 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,873 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,873 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:32,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 487 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1680861332968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,971 DEBUG [Thread-1006] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:54:32,973 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:32,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:32,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:32,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:32,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:32,976 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:32,977 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,977 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:32,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. started... 2023-04-07 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. as already flushing 2023-04-07 09:54:32,980 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:32,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:32,983 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:32,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ea89488[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:32,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:32,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:32,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:32,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:32,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:32,990 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:32,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:32,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:32,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:32,990 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:32,991 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,991 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:32,992 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:32,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:32,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:32,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:32,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:32,996 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:32,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:32,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:32,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:32,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:32,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:32,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:32,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:32,997 DEBUG [Thread-1012] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:32,997 DEBUG [Thread-1012] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:32,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:33,011 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=971, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/ad37e8ad1fa945fea6d6811bdd4c23b0 2023-04-07 09:54:33,034 DEBUG [Thread-1013] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:49660 2023-04-07 09:54:33,034 DEBUG [Thread-1015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:49660 2023-04-07 09:54:33,034 DEBUG [Thread-1013] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:33,034 DEBUG [Thread-1015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:33,036 DEBUG [Thread-1017] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:49660 2023-04-07 09:54:33,037 DEBUG [Thread-1017] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:33,040 DEBUG [Thread-1019] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:49660 2023-04-07 09:54:33,040 DEBUG [Thread-1019] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:33,097 DEBUG [Thread-1012] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:33,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:33,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:33,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58752 deadline: 1680861333309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:33,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:33,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58782 deadline: 1680861333310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:33,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:33,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58814 deadline: 1680861333310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:33,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:33,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58774 deadline: 1680861333311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:33,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=971 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/02ce5583d79c43c1a1886465e79d104e 2023-04-07 09:54:33,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=971 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/7959efa65fb6464fb3ccf2bf5504352c 2023-04-07 09:54:33,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/ad37e8ad1fa945fea6d6811bdd4c23b0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad37e8ad1fa945fea6d6811bdd4c23b0 2023-04-07 09:54:33,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad37e8ad1fa945fea6d6811bdd4c23b0, entries=250, sequenceid=971, filesize=39.2 K 2023-04-07 09:54:33,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/02ce5583d79c43c1a1886465e79d104e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/02ce5583d79c43c1a1886465e79d104e 2023-04-07 09:54:33,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/02ce5583d79c43c1a1886465e79d104e, entries=150, sequenceid=971, filesize=11.9 K 2023-04-07 09:54:33,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/7959efa65fb6464fb3ccf2bf5504352c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7959efa65fb6464fb3ccf2bf5504352c 2023-04-07 09:54:33,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7959efa65fb6464fb3ccf2bf5504352c, entries=150, sequenceid=971, filesize=11.9 K 2023-04-07 09:54:33,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b48a920edd75a7abe23cbbe5f3bc5475 in 1667ms, sequenceid=971, compaction requested=true 2023-04-07 09:54:33,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:33,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:33,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:33,850 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:33,850 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:33,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:33,851 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93074 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:33,851 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:33,852 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/A is initiating minor compaction (all files) 2023-04-07 09:54:33,852 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/B is initiating minor compaction (all files) 2023-04-07 09:54:33,852 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/A in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:33,852 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/B in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:33,852 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/238715ef1b884057bc5a1d8fb6b088e4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/98d18c56fc7243a0a96e72888e9a9e4c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad37e8ad1fa945fea6d6811bdd4c23b0] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=90.9 K 2023-04-07 09:54:33,852 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/5982b5636bdc4278976765a0a40c7b9c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f91ebc597eb64b06a65352486cb36f7b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/02ce5583d79c43c1a1886465e79d104e] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=36.5 K 2023-04-07 09:54:33,852 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 238715ef1b884057bc5a1d8fb6b088e4, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1680861268824 2023-04-07 09:54:33,852 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 5982b5636bdc4278976765a0a40c7b9c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1680861268824 2023-04-07 09:54:33,853 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 98d18c56fc7243a0a96e72888e9a9e4c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1680861269418 2023-04-07 09:54:33,853 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f91ebc597eb64b06a65352486cb36f7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1680861269418 2023-04-07 09:54:33,853 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ad37e8ad1fa945fea6d6811bdd4c23b0, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=971, earliestPutTs=1680861271566 2023-04-07 09:54:33,853 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 02ce5583d79c43c1a1886465e79d104e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=971, earliestPutTs=1680861271570 2023-04-07 09:54:33,859 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#B#compaction#222 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:33,863 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#A#compaction#223 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:34,267 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/0cdc25c8f37d43bc9b97306fed7751f0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0cdc25c8f37d43bc9b97306fed7751f0 2023-04-07 09:54:34,272 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/B of b48a920edd75a7abe23cbbe5f3bc5475 into 0cdc25c8f37d43bc9b97306fed7751f0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:34,272 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:34,273 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/B, priority=13, startTime=1680861273850; duration=0sec 2023-04-07 09:54:34,273 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:34,273 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:34,274 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:34,274 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): b48a920edd75a7abe23cbbe5f3bc5475/C is initiating minor compaction (all files) 2023-04-07 09:54:34,274 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b48a920edd75a7abe23cbbe5f3bc5475/C in TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:34,274 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cc835d58f0b641bc8853b4472a6a64e2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7e0fdd3a9c3d4257a52c55fe1bd50260, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7959efa65fb6464fb3ccf2bf5504352c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp, totalSize=36.5 K 2023-04-07 09:54:34,274 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cc835d58f0b641bc8853b4472a6a64e2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1680861268824 2023-04-07 09:54:34,275 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7e0fdd3a9c3d4257a52c55fe1bd50260, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1680861269418 2023-04-07 09:54:34,275 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7959efa65fb6464fb3ccf2bf5504352c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=971, earliestPutTs=1680861271570 2023-04-07 09:54:34,281 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): b48a920edd75a7abe23cbbe5f3bc5475#C#compaction#224 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:34,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:34,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 09:54:34,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:34,314 DEBUG [Thread-1004] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:49660 2023-04-07 09:54:34,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:34,314 DEBUG [Thread-1004] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:34,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:34,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:34,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:34,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:34,318 DEBUG [Thread-1008] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:49660 2023-04-07 09:54:34,318 DEBUG [Thread-1002] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:49660 2023-04-07 09:54:34,318 DEBUG [Thread-1008] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:34,318 DEBUG [Thread-1002] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:34,322 DEBUG [Thread-1010] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:49660 2023-04-07 09:54:34,322 DEBUG [Thread-1010] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:34,681 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/e9404e8a83f7401082ddd35df665cf91 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e9404e8a83f7401082ddd35df665cf91 2023-04-07 09:54:34,685 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/A of b48a920edd75a7abe23cbbe5f3bc5475 into e9404e8a83f7401082ddd35df665cf91(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:34,685 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:34,685 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/A, priority=13, startTime=1680861273850; duration=0sec 2023-04-07 09:54:34,686 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:34,689 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/a5eb573588e04fcb878bad586c44dbea as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5eb573588e04fcb878bad586c44dbea 2023-04-07 09:54:34,694 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b48a920edd75a7abe23cbbe5f3bc5475/C of b48a920edd75a7abe23cbbe5f3bc5475 into a5eb573588e04fcb878bad586c44dbea(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:34,694 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:34,694 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475., storeName=b48a920edd75a7abe23cbbe5f3bc5475/C, priority=13, startTime=1680861273850; duration=0sec 2023-04-07 09:54:34,694 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:34,728 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040765c17c00ec71464c81c6f7e7fa4c6c5c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040765c17c00ec71464c81c6f7e7fa4c6c5c 2023-04-07 09:54:35,133 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=986, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/a0b05636f5214d28b9792bd35440ee79 2023-04-07 09:54:35,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=986 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/e33d241959fb47a8878ba0f4df56f6ab 2023-04-07 09:54:35,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=986 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/96ac2b5c0cd84c17bc034d9e97e2174c 2023-04-07 09:54:35,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/a0b05636f5214d28b9792bd35440ee79 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/a0b05636f5214d28b9792bd35440ee79 2023-04-07 09:54:35,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/a0b05636f5214d28b9792bd35440ee79, entries=150, sequenceid=986, filesize=25.4 K 2023-04-07 09:54:35,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/e33d241959fb47a8878ba0f4df56f6ab as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/e33d241959fb47a8878ba0f4df56f6ab 2023-04-07 09:54:35,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/e33d241959fb47a8878ba0f4df56f6ab, entries=150, sequenceid=986, filesize=11.9 K 2023-04-07 09:54:35,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/96ac2b5c0cd84c17bc034d9e97e2174c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/96ac2b5c0cd84c17bc034d9e97e2174c 2023-04-07 09:54:35,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/96ac2b5c0cd84c17bc034d9e97e2174c, entries=150, sequenceid=986, filesize=11.9 K 2023-04-07 09:54:35,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=20.13 KB/20610 for b48a920edd75a7abe23cbbe5f3bc5475 in 1651ms, sequenceid=986, compaction requested=false 2023-04-07 09:54:35,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:37,010 DEBUG [Thread-1006] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:49660 2023-04-07 09:54:37,010 DEBUG [Thread-1006] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 251 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 384 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4928 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4722 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 2026 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 6077 rows 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 2036 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 6107 rows 2023-04-07 09:54:37,011 INFO [Listener at localhost/39511] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 09:54:37,011 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:49660 2023-04-07 09:54:37,011 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:54:37,013 INFO [Listener at localhost/39511] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 09:54:37,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 09:54:37,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:37,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 09:54:37,019 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861277018"}]},"ts":"1680861277018"} 2023-04-07 09:54:37,019 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 09:54:37,021 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 09:54:37,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, UNASSIGN}] 2023-04-07 09:54:37,023 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, UNASSIGN 2023-04-07 09:54:37,023 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:37,023 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861277023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861277023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861277023"}]},"ts":"1680861277023"} 2023-04-07 09:54:37,024 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:54:37,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 09:54:37,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b48a920edd75a7abe23cbbe5f3bc5475, disabling compactions & flushes 2023-04-07 09:54:37,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. after waiting 0 ms 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:37,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b48a920edd75a7abe23cbbe5f3bc5475 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=A 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=B 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:37,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b48a920edd75a7abe23cbbe5f3bc5475, store=C 2023-04-07 09:54:37,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:37,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 09:54:37,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407f63d5ebbe2f342849f8fce44f120676e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f63d5ebbe2f342849f8fce44f120676e 2023-04-07 09:54:37,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 09:54:37,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=995, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/2ec63a3307d8415aa8322a990fe9a008 2023-04-07 09:54:38,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 09:54:38,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=995 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/7e4b466b02284bb1847c90f4fb959ca6 2023-04-07 09:54:38,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=995 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/7b34e2df2798422ebc37b93b6dac7544 2023-04-07 09:54:38,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/A/2ec63a3307d8415aa8322a990fe9a008 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2ec63a3307d8415aa8322a990fe9a008 2023-04-07 09:54:38,817 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2ec63a3307d8415aa8322a990fe9a008, entries=100, sequenceid=995, filesize=18.6 K 2023-04-07 09:54:38,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/B/7e4b466b02284bb1847c90f4fb959ca6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7e4b466b02284bb1847c90f4fb959ca6 2023-04-07 09:54:38,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7e4b466b02284bb1847c90f4fb959ca6, entries=100, sequenceid=995, filesize=9.5 K 2023-04-07 09:54:38,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/.tmp/C/7b34e2df2798422ebc37b93b6dac7544 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7b34e2df2798422ebc37b93b6dac7544 2023-04-07 09:54:38,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7b34e2df2798422ebc37b93b6dac7544, entries=100, sequenceid=995, filesize=9.5 K 2023-04-07 09:54:38,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for b48a920edd75a7abe23cbbe5f3bc5475 in 1649ms, sequenceid=995, compaction requested=true 2023-04-07 09:54:38,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1e687068c97f4b928536bbdc47819f32, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/bbfae75e34bb47cbb12c225502957242, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4f746498e4cc410396b698032ca28b0f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d8d808cdca1e479a84c546db1f766538, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/82b05ca10d2642a794640a83dd06f593, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f86bb541447042409b83de822db1157d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/860f1fb710554773a48fdd443e9a1b2b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/17e40c4c53114611877fb6525cb9175d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f9cddf7436fb4cd3be0fd7aaa2803f6c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2b8f76d02cb04949b74d28e62f764698, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e5f6173dbfe34a4e9f377212825baf4d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/b820a6d2015948ccb345690b951230a3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/c6512d87ef104798b247e7c69f96e5c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/de775faaa4cb41d2b49ed043c625069c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52ef6f51fea149df908d4e75136f0222, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/09ced868673d41cea8fed40319f374fc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/aac46a5cd3074c87acc8af9e2342ccab, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/62ea0166b2e0432683094188ba3daea0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad6036e2f4f44bd2ad21d8ed99839061, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4983f7046a5a40e699d182eb3cf1410e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/554c1c56b9e54756b19c83c7133a415c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/7201032c974f434296ec3f3375d1f3c7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1c7a61d543374220afd32c474527afd9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d0550812d5964d15a335d6c3ed48a2d8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52f51b6d02244d56bd5ac989cb7c3890, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/238715ef1b884057bc5a1d8fb6b088e4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/98d18c56fc7243a0a96e72888e9a9e4c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad37e8ad1fa945fea6d6811bdd4c23b0] to archive 2023-04-07 09:54:38,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:54:38,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1e687068c97f4b928536bbdc47819f32 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1e687068c97f4b928536bbdc47819f32 2023-04-07 09:54:38,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/bbfae75e34bb47cbb12c225502957242 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/bbfae75e34bb47cbb12c225502957242 2023-04-07 09:54:38,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4f746498e4cc410396b698032ca28b0f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4f746498e4cc410396b698032ca28b0f 2023-04-07 09:54:38,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d8d808cdca1e479a84c546db1f766538 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d8d808cdca1e479a84c546db1f766538 2023-04-07 09:54:38,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/82b05ca10d2642a794640a83dd06f593 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/82b05ca10d2642a794640a83dd06f593 2023-04-07 09:54:38,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f86bb541447042409b83de822db1157d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f86bb541447042409b83de822db1157d 2023-04-07 09:54:38,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/860f1fb710554773a48fdd443e9a1b2b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/860f1fb710554773a48fdd443e9a1b2b 2023-04-07 09:54:38,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/17e40c4c53114611877fb6525cb9175d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/17e40c4c53114611877fb6525cb9175d 2023-04-07 09:54:38,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f9cddf7436fb4cd3be0fd7aaa2803f6c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/f9cddf7436fb4cd3be0fd7aaa2803f6c 2023-04-07 09:54:38,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2b8f76d02cb04949b74d28e62f764698 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2b8f76d02cb04949b74d28e62f764698 2023-04-07 09:54:38,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e5f6173dbfe34a4e9f377212825baf4d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e5f6173dbfe34a4e9f377212825baf4d 2023-04-07 09:54:38,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/b820a6d2015948ccb345690b951230a3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/b820a6d2015948ccb345690b951230a3 2023-04-07 09:54:38,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/c6512d87ef104798b247e7c69f96e5c1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/c6512d87ef104798b247e7c69f96e5c1 2023-04-07 09:54:38,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/de775faaa4cb41d2b49ed043c625069c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/de775faaa4cb41d2b49ed043c625069c 2023-04-07 09:54:38,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52ef6f51fea149df908d4e75136f0222 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52ef6f51fea149df908d4e75136f0222 2023-04-07 09:54:38,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/09ced868673d41cea8fed40319f374fc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/09ced868673d41cea8fed40319f374fc 2023-04-07 09:54:38,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/aac46a5cd3074c87acc8af9e2342ccab to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/aac46a5cd3074c87acc8af9e2342ccab 2023-04-07 09:54:38,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/62ea0166b2e0432683094188ba3daea0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/62ea0166b2e0432683094188ba3daea0 2023-04-07 09:54:38,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad6036e2f4f44bd2ad21d8ed99839061 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad6036e2f4f44bd2ad21d8ed99839061 2023-04-07 09:54:38,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4983f7046a5a40e699d182eb3cf1410e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/4983f7046a5a40e699d182eb3cf1410e 2023-04-07 09:54:38,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/554c1c56b9e54756b19c83c7133a415c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/554c1c56b9e54756b19c83c7133a415c 2023-04-07 09:54:38,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/7201032c974f434296ec3f3375d1f3c7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/7201032c974f434296ec3f3375d1f3c7 2023-04-07 09:54:38,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1c7a61d543374220afd32c474527afd9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/1c7a61d543374220afd32c474527afd9 2023-04-07 09:54:38,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d0550812d5964d15a335d6c3ed48a2d8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/d0550812d5964d15a335d6c3ed48a2d8 2023-04-07 09:54:38,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52f51b6d02244d56bd5ac989cb7c3890 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/52f51b6d02244d56bd5ac989cb7c3890 2023-04-07 09:54:38,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/238715ef1b884057bc5a1d8fb6b088e4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/238715ef1b884057bc5a1d8fb6b088e4 2023-04-07 09:54:38,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/98d18c56fc7243a0a96e72888e9a9e4c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/98d18c56fc7243a0a96e72888e9a9e4c 2023-04-07 09:54:38,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad37e8ad1fa945fea6d6811bdd4c23b0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/ad37e8ad1fa945fea6d6811bdd4c23b0 2023-04-07 09:54:38,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/fded5e53e3ba4ffc8d0e0be03b0f4c53, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f5b19a9257ff4249aa3c00ba4d8143f5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/90482451bee04a118800c90365298c83, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/b81e4b73edf34abba81b240e177aef77, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/16255d2e05ef49dca3e95a2569569edc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/3ca59e6a67dc46368a3b9417635f8e3d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/8dc910e183f448dd956525d0269597bb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bf45a48a3d2e4462b3641de592dab1c6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a17f8b791f394cf2a3468377f18bb615, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4bf169503b724dfb94723f502ae27eff, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/2fae2b6287924144b85606324c0dcaf5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/9e39fa550672497b8201bf13cb484b48, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/658ac96170734bbeaab79d378ec2b93a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/35864ff4f2024000b5760ee88ccdcab2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/60c940d897b643529e37ea2b0ac45baa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4221353efd7346d6a3f7f7f780ac488f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0606e8d6aee345cdbfcc80d967bf7fad, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/631bd18d77e8449486a3166aed010b2e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bfd0c696ac614ecd8857e67e97954bdd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7d1696648791425bbcb618850a234d32, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/311549c965b849f69404cfdc2bc955f7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a27abd485b4847559a94a81bbfd1233b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/db010813b7c84a00a6adbf25bf0af5ac, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/6fe5737d57ae4228b45a84fba259ee1b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/5982b5636bdc4278976765a0a40c7b9c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/512a6e93c913440c953c3a9ebad7cb75, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f91ebc597eb64b06a65352486cb36f7b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/02ce5583d79c43c1a1886465e79d104e] to archive 2023-04-07 09:54:38,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:54:38,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/fded5e53e3ba4ffc8d0e0be03b0f4c53 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/fded5e53e3ba4ffc8d0e0be03b0f4c53 2023-04-07 09:54:38,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f5b19a9257ff4249aa3c00ba4d8143f5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f5b19a9257ff4249aa3c00ba4d8143f5 2023-04-07 09:54:38,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/90482451bee04a118800c90365298c83 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/90482451bee04a118800c90365298c83 2023-04-07 09:54:38,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/b81e4b73edf34abba81b240e177aef77 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/b81e4b73edf34abba81b240e177aef77 2023-04-07 09:54:38,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/16255d2e05ef49dca3e95a2569569edc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/16255d2e05ef49dca3e95a2569569edc 2023-04-07 09:54:38,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/3ca59e6a67dc46368a3b9417635f8e3d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/3ca59e6a67dc46368a3b9417635f8e3d 2023-04-07 09:54:38,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/8dc910e183f448dd956525d0269597bb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/8dc910e183f448dd956525d0269597bb 2023-04-07 09:54:38,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bf45a48a3d2e4462b3641de592dab1c6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bf45a48a3d2e4462b3641de592dab1c6 2023-04-07 09:54:38,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a17f8b791f394cf2a3468377f18bb615 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a17f8b791f394cf2a3468377f18bb615 2023-04-07 09:54:38,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4bf169503b724dfb94723f502ae27eff to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4bf169503b724dfb94723f502ae27eff 2023-04-07 09:54:38,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/2fae2b6287924144b85606324c0dcaf5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/2fae2b6287924144b85606324c0dcaf5 2023-04-07 09:54:38,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/9e39fa550672497b8201bf13cb484b48 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/9e39fa550672497b8201bf13cb484b48 2023-04-07 09:54:38,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/658ac96170734bbeaab79d378ec2b93a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/658ac96170734bbeaab79d378ec2b93a 2023-04-07 09:54:38,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/35864ff4f2024000b5760ee88ccdcab2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/35864ff4f2024000b5760ee88ccdcab2 2023-04-07 09:54:38,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/60c940d897b643529e37ea2b0ac45baa to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/60c940d897b643529e37ea2b0ac45baa 2023-04-07 09:54:38,898 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4221353efd7346d6a3f7f7f780ac488f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/4221353efd7346d6a3f7f7f780ac488f 2023-04-07 09:54:38,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0606e8d6aee345cdbfcc80d967bf7fad to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0606e8d6aee345cdbfcc80d967bf7fad 2023-04-07 09:54:38,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/631bd18d77e8449486a3166aed010b2e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/631bd18d77e8449486a3166aed010b2e 2023-04-07 09:54:38,901 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bfd0c696ac614ecd8857e67e97954bdd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/bfd0c696ac614ecd8857e67e97954bdd 2023-04-07 09:54:38,902 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7d1696648791425bbcb618850a234d32 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7d1696648791425bbcb618850a234d32 2023-04-07 09:54:38,903 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/311549c965b849f69404cfdc2bc955f7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/311549c965b849f69404cfdc2bc955f7 2023-04-07 09:54:38,904 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a27abd485b4847559a94a81bbfd1233b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/a27abd485b4847559a94a81bbfd1233b 2023-04-07 09:54:38,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/db010813b7c84a00a6adbf25bf0af5ac to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/db010813b7c84a00a6adbf25bf0af5ac 2023-04-07 09:54:38,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/6fe5737d57ae4228b45a84fba259ee1b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/6fe5737d57ae4228b45a84fba259ee1b 2023-04-07 09:54:38,906 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/5982b5636bdc4278976765a0a40c7b9c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/5982b5636bdc4278976765a0a40c7b9c 2023-04-07 09:54:38,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/512a6e93c913440c953c3a9ebad7cb75 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/512a6e93c913440c953c3a9ebad7cb75 2023-04-07 09:54:38,909 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f91ebc597eb64b06a65352486cb36f7b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/f91ebc597eb64b06a65352486cb36f7b 2023-04-07 09:54:38,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/02ce5583d79c43c1a1886465e79d104e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/02ce5583d79c43c1a1886465e79d104e 2023-04-07 09:54:38,912 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/b934c1e3857a4b88a20e36e765562848, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/876c86b04a9648e0923d603e4a2238ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c7b01ee857314135a1410fa420b75d2b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5cc80d888b94b3d860e3863b3ce358e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/0f1b6046071c42548fc4685e8f03607a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/424c9057d7674898b8f787258e3a6bab, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/4d80871f7afb450ea61078a46b87e354, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/416a67a1bdeb4d87861c06944a064709, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/6705cba5a2804ebcbc1c2e65204c7714, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/3b92f5a8e7ac496593851640016986dc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/8379fe636cb14d8cb808424b6f355d27, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c3472f1e8a6d4608a8aeaadfdabb2cd2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/31b552e992c04a1e801d8cd65e5987cf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/efa25e40186a479a90748cf202c0b3de, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cf5c15f7b65249259a015449ae0b397c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/015dce71e0284ec48cba68f1e67def14, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a44ad1edc53a4fba95bb72fd155d82e2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a9e3e4e5f6a344b89b380e62f3798572, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/2147c77c4c76453e98c7c10083daafef, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/e479b069286940eca05e9f28702353c2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/94a87531ce6d4cc498347e69e7315187, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/1f844a8663b546f9a39eee7f13297a8a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a16e11998258484eb7fdd7da38533e42, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/006b96a2c7134de0a86fb3f706518e1b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cc835d58f0b641bc8853b4472a6a64e2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/71967e1334c5466ea71b9b27d9274753, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7e0fdd3a9c3d4257a52c55fe1bd50260, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7959efa65fb6464fb3ccf2bf5504352c] to archive 2023-04-07 09:54:38,913 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:54:38,914 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/b934c1e3857a4b88a20e36e765562848 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/b934c1e3857a4b88a20e36e765562848 2023-04-07 09:54:38,915 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/876c86b04a9648e0923d603e4a2238ed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/876c86b04a9648e0923d603e4a2238ed 2023-04-07 09:54:38,916 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c7b01ee857314135a1410fa420b75d2b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c7b01ee857314135a1410fa420b75d2b 2023-04-07 09:54:38,917 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5cc80d888b94b3d860e3863b3ce358e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5cc80d888b94b3d860e3863b3ce358e 2023-04-07 09:54:38,918 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/0f1b6046071c42548fc4685e8f03607a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/0f1b6046071c42548fc4685e8f03607a 2023-04-07 09:54:38,918 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/424c9057d7674898b8f787258e3a6bab to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/424c9057d7674898b8f787258e3a6bab 2023-04-07 09:54:38,919 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/4d80871f7afb450ea61078a46b87e354 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/4d80871f7afb450ea61078a46b87e354 2023-04-07 09:54:38,920 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/416a67a1bdeb4d87861c06944a064709 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/416a67a1bdeb4d87861c06944a064709 2023-04-07 09:54:38,921 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/6705cba5a2804ebcbc1c2e65204c7714 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/6705cba5a2804ebcbc1c2e65204c7714 2023-04-07 09:54:38,922 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/3b92f5a8e7ac496593851640016986dc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/3b92f5a8e7ac496593851640016986dc 2023-04-07 09:54:38,923 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/8379fe636cb14d8cb808424b6f355d27 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/8379fe636cb14d8cb808424b6f355d27 2023-04-07 09:54:38,924 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c3472f1e8a6d4608a8aeaadfdabb2cd2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/c3472f1e8a6d4608a8aeaadfdabb2cd2 2023-04-07 09:54:38,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/31b552e992c04a1e801d8cd65e5987cf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/31b552e992c04a1e801d8cd65e5987cf 2023-04-07 09:54:38,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/efa25e40186a479a90748cf202c0b3de to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/efa25e40186a479a90748cf202c0b3de 2023-04-07 09:54:38,926 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cf5c15f7b65249259a015449ae0b397c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cf5c15f7b65249259a015449ae0b397c 2023-04-07 09:54:38,927 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/015dce71e0284ec48cba68f1e67def14 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/015dce71e0284ec48cba68f1e67def14 2023-04-07 09:54:38,928 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a44ad1edc53a4fba95bb72fd155d82e2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a44ad1edc53a4fba95bb72fd155d82e2 2023-04-07 09:54:38,929 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a9e3e4e5f6a344b89b380e62f3798572 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a9e3e4e5f6a344b89b380e62f3798572 2023-04-07 09:54:38,930 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/2147c77c4c76453e98c7c10083daafef to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/2147c77c4c76453e98c7c10083daafef 2023-04-07 09:54:38,930 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/e479b069286940eca05e9f28702353c2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/e479b069286940eca05e9f28702353c2 2023-04-07 09:54:38,931 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/94a87531ce6d4cc498347e69e7315187 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/94a87531ce6d4cc498347e69e7315187 2023-04-07 09:54:38,932 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/1f844a8663b546f9a39eee7f13297a8a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/1f844a8663b546f9a39eee7f13297a8a 2023-04-07 09:54:38,933 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a16e11998258484eb7fdd7da38533e42 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a16e11998258484eb7fdd7da38533e42 2023-04-07 09:54:38,934 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/006b96a2c7134de0a86fb3f706518e1b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/006b96a2c7134de0a86fb3f706518e1b 2023-04-07 09:54:38,935 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cc835d58f0b641bc8853b4472a6a64e2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/cc835d58f0b641bc8853b4472a6a64e2 2023-04-07 09:54:38,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/71967e1334c5466ea71b9b27d9274753 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/71967e1334c5466ea71b9b27d9274753 2023-04-07 09:54:38,937 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7e0fdd3a9c3d4257a52c55fe1bd50260 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7e0fdd3a9c3d4257a52c55fe1bd50260 2023-04-07 09:54:38,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7959efa65fb6464fb3ccf2bf5504352c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7959efa65fb6464fb3ccf2bf5504352c 2023-04-07 09:54:38,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/recovered.edits/998.seqid, newMaxSeqId=998, maxSeqId=4 2023-04-07 09:54:38,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475. 2023-04-07 09:54:38,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b48a920edd75a7abe23cbbe5f3bc5475: 2023-04-07 09:54:38,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:38,944 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b48a920edd75a7abe23cbbe5f3bc5475, regionState=CLOSED 2023-04-07 09:54:38,945 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861278944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861278944"}]},"ts":"1680861278944"} 2023-04-07 09:54:38,946 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-07 09:54:38,947 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure b48a920edd75a7abe23cbbe5f3bc5475, server=jenkins-hbase4.apache.org,35881,1680861218288 in 1.9210 sec 2023-04-07 09:54:38,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-07 09:54:38,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b48a920edd75a7abe23cbbe5f3bc5475, UNASSIGN in 1.9250 sec 2023-04-07 09:54:38,948 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861278948"}]},"ts":"1680861278948"} 2023-04-07 09:54:38,949 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 09:54:38,950 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 09:54:38,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9380 sec 2023-04-07 09:54:39,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-07 09:54:39,122 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-07 09:54:39,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 09:54:39,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:39,125 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:39,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-07 09:54:39,126 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:39,131 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:39,133 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/recovered.edits] 2023-04-07 09:54:39,135 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2ec63a3307d8415aa8322a990fe9a008 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/2ec63a3307d8415aa8322a990fe9a008 2023-04-07 09:54:39,137 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/a0b05636f5214d28b9792bd35440ee79 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/a0b05636f5214d28b9792bd35440ee79 2023-04-07 09:54:39,138 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e9404e8a83f7401082ddd35df665cf91 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/A/e9404e8a83f7401082ddd35df665cf91 2023-04-07 09:54:39,140 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0cdc25c8f37d43bc9b97306fed7751f0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/0cdc25c8f37d43bc9b97306fed7751f0 2023-04-07 09:54:39,141 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7e4b466b02284bb1847c90f4fb959ca6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/7e4b466b02284bb1847c90f4fb959ca6 2023-04-07 09:54:39,142 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/e33d241959fb47a8878ba0f4df56f6ab to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/B/e33d241959fb47a8878ba0f4df56f6ab 2023-04-07 09:54:39,144 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7b34e2df2798422ebc37b93b6dac7544 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/7b34e2df2798422ebc37b93b6dac7544 2023-04-07 09:54:39,145 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/96ac2b5c0cd84c17bc034d9e97e2174c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/96ac2b5c0cd84c17bc034d9e97e2174c 2023-04-07 09:54:39,146 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5eb573588e04fcb878bad586c44dbea to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/C/a5eb573588e04fcb878bad586c44dbea 2023-04-07 09:54:39,148 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/recovered.edits/998.seqid to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475/recovered.edits/998.seqid 2023-04-07 09:54:39,149 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/b48a920edd75a7abe23cbbe5f3bc5475 2023-04-07 09:54:39,149 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:54:39,150 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 09:54:39,150 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-07 09:54:39,154 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040701d97d82c51c4098a208d922ddb5c8f3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040701d97d82c51c4098a208d922ddb5c8f3 2023-04-07 09:54:39,155 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407087011828c89427284b59babae9674e0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407087011828c89427284b59babae9674e0 2023-04-07 09:54:39,156 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070a23d0d4fc164699b884d6ece9475aee to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070a23d0d4fc164699b884d6ece9475aee 2023-04-07 09:54:39,157 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040727bd71bdd3d640949af1a1e4627cb126 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040727bd71bdd3d640949af1a1e4627cb126 2023-04-07 09:54:39,158 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072f1e63131a85460f8a0dad0c2baef31a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304072f1e63131a85460f8a0dad0c2baef31a 2023-04-07 09:54:39,159 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040757337cb643b44c4d90df52c53b66a173 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040757337cb643b44c4d90df52c53b66a173 2023-04-07 09:54:39,160 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040759dfabb32aa240f1aed7d44231ca428f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040759dfabb32aa240f1aed7d44231ca428f 2023-04-07 09:54:39,161 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040765c17c00ec71464c81c6f7e7fa4c6c5c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040765c17c00ec71464c81c6f7e7fa4c6c5c 2023-04-07 09:54:39,162 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040767dede6043ab4bf98e2d7c2692610627 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040767dede6043ab4bf98e2d7c2692610627 2023-04-07 09:54:39,163 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076c73be90670249798c5494e9a9e27df1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304076c73be90670249798c5494e9a9e27df1 2023-04-07 09:54:39,164 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407729e96151cce40bea09a3b155c3ea566 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407729e96151cce40bea09a3b155c3ea566 2023-04-07 09:54:39,165 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077ed04ba57e894dc9a2b017c8ae13ddfe to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077ed04ba57e894dc9a2b017c8ae13ddfe 2023-04-07 09:54:39,166 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040792421baa8c9945f2875cf3f7f286bf9a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040792421baa8c9945f2875cf3f7f286bf9a 2023-04-07 09:54:39,167 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a1e2acd689a143c1be0744ccd154228d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a1e2acd689a143c1be0744ccd154228d 2023-04-07 09:54:39,168 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a3cb3e711d36446897c671282f7efa4f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a3cb3e711d36446897c671282f7efa4f 2023-04-07 09:54:39,169 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a65275748ad64bee83f132b0f33001ca to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a65275748ad64bee83f132b0f33001ca 2023-04-07 09:54:39,170 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b2ec243b57544ab59b3ef2eb2058a3b1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b2ec243b57544ab59b3ef2eb2058a3b1 2023-04-07 09:54:39,171 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d5a1fe42dfd34e55b95eba6d177382a1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d5a1fe42dfd34e55b95eba6d177382a1 2023-04-07 09:54:39,172 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e5dd8f663e7e48fb89ace33f42f7fb61 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e5dd8f663e7e48fb89ace33f42f7fb61 2023-04-07 09:54:39,173 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e86a7820293a475b93e7a6d56ac486be to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e86a7820293a475b93e7a6d56ac486be 2023-04-07 09:54:39,174 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f63d5ebbe2f342849f8fce44f120676e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f63d5ebbe2f342849f8fce44f120676e 2023-04-07 09:54:39,175 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fdc2bd28db7649ae8ec61c1583fdab62 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fdc2bd28db7649ae8ec61c1583fdab62 2023-04-07 09:54:39,176 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fe213b99e9b24e6b9b2c140f500b72f6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fe213b99e9b24e6b9b2c140f500b72f6 2023-04-07 09:54:39,176 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 09:54:39,179 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:39,180 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 09:54:39,182 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 09:54:39,183 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:39,183 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 09:54:39,183 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680861279183"}]},"ts":"9223372036854775807"} 2023-04-07 09:54:39,185 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 09:54:39,185 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b48a920edd75a7abe23cbbe5f3bc5475, NAME => 'TestAcidGuarantees,,1680861250242.b48a920edd75a7abe23cbbe5f3bc5475.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 09:54:39,185 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 09:54:39,185 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680861279185"}]},"ts":"9223372036854775807"} 2023-04-07 09:54:39,186 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 09:54:39,188 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:39,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 65 msec 2023-04-07 09:54:39,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-07 09:54:39,227 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-07 09:54:39,239 INFO [Listener at localhost/39511] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233 (was 235), OpenFileDescriptor=558 (was 593), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=434 (was 412) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=2810 (was 3021) 2023-04-07 09:54:39,250 INFO [Listener at localhost/39511] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=233, OpenFileDescriptor=558, MaxFileDescriptor=60000, SystemLoadAverage=434, ProcessCount=169, AvailableMemoryMB=2810 2023-04-07 09:54:39,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:54:39,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:54:39,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 09:54:39,255 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 09:54:39,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-07 09:54:39,256 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 09:54:39,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 09:54:39,258 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:39,258 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90 empty. 2023-04-07 09:54:39,259 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:39,259 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:54:39,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 09:54:39,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 09:54:39,667 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:54:39,669 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 243bf66836bfdaa40ee21a1c79eaec90, NAME => 'TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp 2023-04-07 09:54:39,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 09:54:40,074 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:54:40,074 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 243bf66836bfdaa40ee21a1c79eaec90, disabling compactions & flushes 2023-04-07 09:54:40,074 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,074 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,074 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. after waiting 0 ms 2023-04-07 09:54:40,074 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,075 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,075 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:40,077 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 09:54:40,078 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861280077"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861280077"}]},"ts":"1680861280077"} 2023-04-07 09:54:40,078 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 09:54:40,079 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 09:54:40,079 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861280079"}]},"ts":"1680861280079"} 2023-04-07 09:54:40,080 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 09:54:40,082 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=243bf66836bfdaa40ee21a1c79eaec90, ASSIGN}] 2023-04-07 09:54:40,083 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=243bf66836bfdaa40ee21a1c79eaec90, ASSIGN 2023-04-07 09:54:40,084 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=243bf66836bfdaa40ee21a1c79eaec90, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=false 2023-04-07 09:54:40,235 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=243bf66836bfdaa40ee21a1c79eaec90, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:40,236 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861280235"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861280235"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861280235"}]},"ts":"1680861280235"} 2023-04-07 09:54:40,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:54:40,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 09:54:40,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 243bf66836bfdaa40ee21a1c79eaec90, NAME => 'TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:54:40,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:54:40,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,394 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,396 DEBUG [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A 2023-04-07 09:54:40,396 DEBUG [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A 2023-04-07 09:54:40,396 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:40,396 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 243bf66836bfdaa40ee21a1c79eaec90 columnFamilyName A 2023-04-07 09:54:40,397 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.HStore(310): Store=243bf66836bfdaa40ee21a1c79eaec90/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:40,397 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,398 DEBUG [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B 2023-04-07 09:54:40,398 DEBUG [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B 2023-04-07 09:54:40,398 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:40,398 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 243bf66836bfdaa40ee21a1c79eaec90 columnFamilyName B 2023-04-07 09:54:40,399 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.HStore(310): Store=243bf66836bfdaa40ee21a1c79eaec90/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:40,399 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,400 DEBUG [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C 2023-04-07 09:54:40,400 DEBUG [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C 2023-04-07 09:54:40,400 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:54:40,400 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 243bf66836bfdaa40ee21a1c79eaec90 columnFamilyName C 2023-04-07 09:54:40,401 INFO [StoreOpener-243bf66836bfdaa40ee21a1c79eaec90-1] regionserver.HStore(310): Store=243bf66836bfdaa40ee21a1c79eaec90/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:54:40,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,403 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-07 09:54:40,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:54:40,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:40,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:54:40,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 243bf66836bfdaa40ee21a1c79eaec90; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72511956, jitterRate=0.08051234483718872}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:54:40,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:40,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., pid=30, masterSystemTime=1680861280389 2023-04-07 09:54:40,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:40,408 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=243bf66836bfdaa40ee21a1c79eaec90, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:40,409 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861280408"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861280408"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861280408"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861280408"}]},"ts":"1680861280408"} 2023-04-07 09:54:40,411 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-07 09:54:40,411 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 in 172 msec 2023-04-07 09:54:40,412 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-07 09:54:40,412 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=243bf66836bfdaa40ee21a1c79eaec90, ASSIGN in 329 msec 2023-04-07 09:54:40,413 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 09:54:40,413 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861280413"}]},"ts":"1680861280413"} 2023-04-07 09:54:40,413 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 09:54:40,416 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 09:54:40,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1630 sec 2023-04-07 09:54:41,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-07 09:54:41,361 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-07 09:54:41,362 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,366 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,367 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,369 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,370 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 09:54:41,372 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 09:54:41,373 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,376 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,377 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,380 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,380 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,385 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,385 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,388 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,388 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,393 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,394 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,399 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,399 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,408 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,408 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,412 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,413 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,416 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,417 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:54:41,420 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:54:41,422 DEBUG [hconnection-0x6a011b99-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,422 DEBUG [hconnection-0x2bf04f72-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,423 DEBUG [hconnection-0x20f74686-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,423 DEBUG [hconnection-0x44fbd99f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,424 DEBUG [hconnection-0x792dcfe0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,429 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:41,431 DEBUG [hconnection-0x1a5a2a04-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,433 DEBUG [hconnection-0x4aae601a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,432 DEBUG [hconnection-0x605d0e06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,432 DEBUG [hconnection-0x1f4f9ee8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,435 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48264, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,435 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,435 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,434 DEBUG [hconnection-0x2b2eae47-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:54:41,438 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,436 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48308, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,438 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48294, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,439 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,441 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,441 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:54:41,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:41,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:41,445 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:41,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:41,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:41,447 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,447 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:41,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:41,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:41,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:41,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:41,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:54:41,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:41,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:41,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:41,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:41,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:41,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:41,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:41,473 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,473 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:41,473 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:41,473 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:41,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:41,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50bf01c7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:41,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:41,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,478 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:41,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:41,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861341477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:41,480 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/929ce8a91ca74e77ba347f5baf161a46 2023-04-07 09:54:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861341479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861341480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861341481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:41,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:41,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:41,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:41,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861341481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:41,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:41,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:41,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:41,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,493 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:41,493 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:41,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:41,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861341581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861341584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861341585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861341585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861341586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,593 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:41,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:41,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:41,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:41,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:41,596 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:41,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:41,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:41,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:41,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:41,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:41,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:41,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:41,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:41,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:41,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,604 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:41,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:41,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a742d6a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:41,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:41,608 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:41,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:41,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:41,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:41,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:41,613 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:41,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:41,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:41,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:41,613 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:41,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,620 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:41,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:41,621 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,621 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:41,621 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:41,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,721 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:41,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:41,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:41,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:41,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:41,724 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:41,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:41,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:41,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,725 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,725 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:41,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:41,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:41,727 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,727 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:41,727 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:41,727 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:41,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:41,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:41,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:41,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,729 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:41,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:41,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43bc0b69[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:41,732 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:41,732 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,734 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:41,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:41,736 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:41,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:41,740 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:41,740 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:41,740 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:41,740 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:41,740 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:41,741 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,742 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,742 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,743 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,744 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:41,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:41,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,747 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:41,747 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:41,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:41,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861341784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861341786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861341787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861341788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:41,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861341788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,847 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:41,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:41,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:41,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:41,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:41,850 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:41,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:41,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:41,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:41,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:41,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:41,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:41,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:41,854 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:41,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:41,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:41,855 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:41,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:41,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:41,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@295adb42[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:41,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:41,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:41,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:41,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:41,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:41,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:41,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:41,866 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:41,866 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:41,866 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:41,866 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:41,866 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:41,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:41,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:41,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:41,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:41,872 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:41,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:41,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:41,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,872 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:41,872 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:41,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:41,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/a1dff585c7ec40c298bdb86ade929f31 2023-04-07 09:54:41,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/82864fc02fcc4e4e946f9ccdcc5d1cf4 2023-04-07 09:54:41,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/929ce8a91ca74e77ba347f5baf161a46 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/929ce8a91ca74e77ba347f5baf161a46 2023-04-07 09:54:41,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/929ce8a91ca74e77ba347f5baf161a46, entries=150, sequenceid=15, filesize=11.6 K 2023-04-07 09:54:41,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/a1dff585c7ec40c298bdb86ade929f31 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a1dff585c7ec40c298bdb86ade929f31 2023-04-07 09:54:41,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a1dff585c7ec40c298bdb86ade929f31, entries=150, sequenceid=15, filesize=11.6 K 2023-04-07 09:54:41,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/82864fc02fcc4e4e946f9ccdcc5d1cf4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/82864fc02fcc4e4e946f9ccdcc5d1cf4 2023-04-07 09:54:41,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/82864fc02fcc4e4e946f9ccdcc5d1cf4, entries=150, sequenceid=15, filesize=11.6 K 2023-04-07 09:54:41,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 243bf66836bfdaa40ee21a1c79eaec90 in 482ms, sequenceid=15, compaction requested=false 2023-04-07 09:54:41,941 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-07 09:54:41,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:41,972 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:41,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:41,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:41,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:41,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:41,980 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:41,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:41,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:41,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:41,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:41,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:41,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:41,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:41,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:41,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:41,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:41,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:41,992 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:41,992 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:41,992 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:41,996 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:41,996 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:41,996 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:41,996 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:41,997 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:54:41,997 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:41,998 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:41,998 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:41,998 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:41,998 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:41,998 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:42,034 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/8d060c8b1e964e228477d57e7da6dbdb 2023-04-07 09:54:42,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:42,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:42,091 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/7ba8e5380515496f807b40fbfca9a653 2023-04-07 09:54:42,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861342097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861342097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861342098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861342099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861342100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861342203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861342203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861342203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861342204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861342204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861342406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861342406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861342407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861342407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861342408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,535 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/9811e6b56cc549369fd93275e061c26f 2023-04-07 09:54:42,539 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/8d060c8b1e964e228477d57e7da6dbdb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/8d060c8b1e964e228477d57e7da6dbdb 2023-04-07 09:54:42,542 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/8d060c8b1e964e228477d57e7da6dbdb, entries=150, sequenceid=34, filesize=11.6 K 2023-04-07 09:54:42,543 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/7ba8e5380515496f807b40fbfca9a653 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7ba8e5380515496f807b40fbfca9a653 2023-04-07 09:54:42,547 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7ba8e5380515496f807b40fbfca9a653, entries=150, sequenceid=34, filesize=11.6 K 2023-04-07 09:54:42,547 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/9811e6b56cc549369fd93275e061c26f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/9811e6b56cc549369fd93275e061c26f 2023-04-07 09:54:42,550 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/9811e6b56cc549369fd93275e061c26f, entries=150, sequenceid=34, filesize=11.6 K 2023-04-07 09:54:42,551 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 243bf66836bfdaa40ee21a1c79eaec90 in 554ms, sequenceid=34, compaction requested=false 2023-04-07 09:54:42,552 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:42,552 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:42,554 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:42,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1434697[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:42,556 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:42,556 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,557 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:42,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:42,557 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,557 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:42,558 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:42,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:42,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:42,561 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:42,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:42,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:42,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:42,561 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:42,563 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,563 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,563 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:42,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,565 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:42,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:42,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:42,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:42,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:42,568 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,568 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:42,669 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:42,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:42,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:42,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:42,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:42,672 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:42,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:42,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:42,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:42,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,674 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:42,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:42,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:42,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:42,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:42,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:42,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:42,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:42,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:42,676 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:42,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:42,683 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/936aa60d358d423fbaf46f12fb68d5e9 2023-04-07 09:54:42,692 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/78c37007227e478e9169673bb8b18e1a 2023-04-07 09:54:42,702 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/dfd51d4f8073414da2bd824c0884b2a2 2023-04-07 09:54:42,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/936aa60d358d423fbaf46f12fb68d5e9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/936aa60d358d423fbaf46f12fb68d5e9 2023-04-07 09:54:42,710 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/936aa60d358d423fbaf46f12fb68d5e9, entries=150, sequenceid=47, filesize=11.6 K 2023-04-07 09:54:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:42,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:42,712 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/78c37007227e478e9169673bb8b18e1a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/78c37007227e478e9169673bb8b18e1a 2023-04-07 09:54:42,722 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/78c37007227e478e9169673bb8b18e1a, entries=150, sequenceid=47, filesize=11.6 K 2023-04-07 09:54:42,723 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/dfd51d4f8073414da2bd824c0884b2a2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dfd51d4f8073414da2bd824c0884b2a2 2023-04-07 09:54:42,729 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dfd51d4f8073414da2bd824c0884b2a2, entries=150, sequenceid=47, filesize=11.6 K 2023-04-07 09:54:42,730 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=100.63 KB/103050 for 243bf66836bfdaa40ee21a1c79eaec90 in 54ms, sequenceid=47, compaction requested=true 2023-04-07 09:54:42,730 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:42,730 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:42,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:42,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:42,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:42,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:42,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:42,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:54:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:42,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:42,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:42,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2af724a8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:42,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:42,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,741 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:42,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:42,742 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:42,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:42,745 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:42,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:42,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:42,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:42,745 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:42,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861342742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,746 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,746 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,746 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:42,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861342743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861342743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:42,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861342743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861342744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:42,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:42,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:42,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,751 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:42,751 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:42,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:42,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861342847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861342848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861342848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861342848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:42,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861342849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,851 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:42,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:42,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:42,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:42,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:42,854 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:42,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:42,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:42,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:42,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,860 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,860 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:42,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:42,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:42,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:42,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:42,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:42,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:42,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:42,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:42,864 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:42,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b395dc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:42,866 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:42,866 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:42,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:42,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:42,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:42,868 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:42,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:42,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:42,871 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:42,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:42,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:42,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:42,871 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:42,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,875 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:42,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:42,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:42,879 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,879 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,879 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:42,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:42,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:42,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:42,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:42,980 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:42,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:42,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:42,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:42,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:42,983 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:42,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:42,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:42,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:42,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:42,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:42,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:42,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:42,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:42,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:42,986 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:42,986 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:42,986 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:42,986 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:42,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:42,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:42,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,990 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:42,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@764cdf63[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:42,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,992 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:42,992 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:42,993 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:42,993 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:42,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:42,994 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:42,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:42,997 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:42,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:42,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:42,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:42,997 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:42,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:42,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:42,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:42,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:43,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:43,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:43,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:43,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:43,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:43,004 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:43,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:43,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:43,005 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:43,005 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:43,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861343049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861343050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861343051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861343051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861343051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,105 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:43,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:43,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:43,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:43,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:43,108 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:43,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:43,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:43,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:43,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:43,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:43,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:43,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:43,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:43,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:43,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:43,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:43,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:43,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d2058c3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:43,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:43,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,117 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,117 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:43,117 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:43,117 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:43,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:43,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:43,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:43,123 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:43,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:43,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:43,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:43,124 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:43,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:43,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:43,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:43,126 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:43,129 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:43,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:43,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:43,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:43,130 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:43,130 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:43,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:43,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3d9f2bf71d57407dac4253c92aea9a3f 2023-04-07 09:54:43,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/2d39eaecbaa14c8d8c651e6a3cdaf5de 2023-04-07 09:54:43,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/3d2fdcfe41894948b373a970d284aeee 2023-04-07 09:54:43,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3d9f2bf71d57407dac4253c92aea9a3f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3d9f2bf71d57407dac4253c92aea9a3f 2023-04-07 09:54:43,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3d9f2bf71d57407dac4253c92aea9a3f, entries=300, sequenceid=67, filesize=18.5 K 2023-04-07 09:54:43,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/2d39eaecbaa14c8d8c651e6a3cdaf5de as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2d39eaecbaa14c8d8c651e6a3cdaf5de 2023-04-07 09:54:43,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2d39eaecbaa14c8d8c651e6a3cdaf5de, entries=150, sequenceid=67, filesize=11.6 K 2023-04-07 09:54:43,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/3d2fdcfe41894948b373a970d284aeee as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3d2fdcfe41894948b373a970d284aeee 2023-04-07 09:54:43,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3d2fdcfe41894948b373a970d284aeee, entries=150, sequenceid=67, filesize=11.6 K 2023-04-07 09:54:43,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 243bf66836bfdaa40ee21a1c79eaec90 in 445ms, sequenceid=67, compaction requested=true 2023-04-07 09:54:43,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:43,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:43,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:43,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:43,176 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:43,176 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:43,179 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54564 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:43,179 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:43,179 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:43,179 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:43,179 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:43,179 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:43,179 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/929ce8a91ca74e77ba347f5baf161a46, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/8d060c8b1e964e228477d57e7da6dbdb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/936aa60d358d423fbaf46f12fb68d5e9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3d9f2bf71d57407dac4253c92aea9a3f] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=53.3 K 2023-04-07 09:54:43,179 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a1dff585c7ec40c298bdb86ade929f31, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7ba8e5380515496f807b40fbfca9a653, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/78c37007227e478e9169673bb8b18e1a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2d39eaecbaa14c8d8c651e6a3cdaf5de] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=46.4 K 2023-04-07 09:54:43,179 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 929ce8a91ca74e77ba347f5baf161a46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680861281448 2023-04-07 09:54:43,179 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a1dff585c7ec40c298bdb86ade929f31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680861281448 2023-04-07 09:54:43,180 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8d060c8b1e964e228477d57e7da6dbdb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680861281478 2023-04-07 09:54:43,180 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7ba8e5380515496f807b40fbfca9a653, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680861281478 2023-04-07 09:54:43,180 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 936aa60d358d423fbaf46f12fb68d5e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680861282096 2023-04-07 09:54:43,180 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 78c37007227e478e9169673bb8b18e1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680861282096 2023-04-07 09:54:43,181 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3d9f2bf71d57407dac4253c92aea9a3f, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680861282722 2023-04-07 09:54:43,181 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2d39eaecbaa14c8d8c651e6a3cdaf5de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680861282729 2023-04-07 09:54:43,189 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#244 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:43,189 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#243 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:43,198 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/e988fe4ab3a74b19b65d3daaae5526cd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e988fe4ab3a74b19b65d3daaae5526cd 2023-04-07 09:54:43,200 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/082da09d8fe344d189e06a0d09beacce as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/082da09d8fe344d189e06a0d09beacce 2023-04-07 09:54:43,204 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into e988fe4ab3a74b19b65d3daaae5526cd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:43,204 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:43,204 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=12, startTime=1680861283176; duration=0sec 2023-04-07 09:54:43,204 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:43,204 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:43,208 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:43,208 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:43,209 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:43,209 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/82864fc02fcc4e4e946f9ccdcc5d1cf4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/9811e6b56cc549369fd93275e061c26f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dfd51d4f8073414da2bd824c0884b2a2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3d2fdcfe41894948b373a970d284aeee] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=46.4 K 2023-04-07 09:54:43,210 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 082da09d8fe344d189e06a0d09beacce(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:43,210 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:43,210 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=12, startTime=1680861283176; duration=0sec 2023-04-07 09:54:43,210 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:43,212 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 82864fc02fcc4e4e946f9ccdcc5d1cf4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680861281448 2023-04-07 09:54:43,213 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9811e6b56cc549369fd93275e061c26f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680861281478 2023-04-07 09:54:43,213 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting dfd51d4f8073414da2bd824c0884b2a2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680861282096 2023-04-07 09:54:43,213 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3d2fdcfe41894948b373a970d284aeee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680861282729 2023-04-07 09:54:43,220 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#245 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:43,230 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:43,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:43,231 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/3970929a211a489bb8e0d603f394846d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3970929a211a489bb8e0d603f394846d 2023-04-07 09:54:43,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:43,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:43,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:43,234 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:43,234 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:43,234 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:43,234 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:43,234 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,234 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:43,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:43,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:43,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:43,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:43,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:43,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:43,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:43,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:43,238 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 3970929a211a489bb8e0d603f394846d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:43,238 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:43,238 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=12, startTime=1680861283176; duration=0sec 2023-04-07 09:54:43,238 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:43,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:43,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:43,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:43,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:43,238 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:54:43,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:43,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:43,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:43,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:43,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:43,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:43,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:43,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:43,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861343378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861343394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861343394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861343394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861343395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861343497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861343498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861343499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861343499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861343499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,648 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3dcdc94da10a4e4dad3223e1791e50fd 2023-04-07 09:54:43,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861343700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861343702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861343702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861343702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:43,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:43,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861343703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861344002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861344004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861344004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861344005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861344006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,061 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/a4319a3905c54026ae42b6ca33d9faf9 2023-04-07 09:54:44,472 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/13454b85e08d4f47a59356c4880ab546 2023-04-07 09:54:44,478 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3dcdc94da10a4e4dad3223e1791e50fd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3dcdc94da10a4e4dad3223e1791e50fd 2023-04-07 09:54:44,482 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 09:54:44,485 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3dcdc94da10a4e4dad3223e1791e50fd, entries=150, sequenceid=83, filesize=11.6 K 2023-04-07 09:54:44,485 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/a4319a3905c54026ae42b6ca33d9faf9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4319a3905c54026ae42b6ca33d9faf9 2023-04-07 09:54:44,495 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4319a3905c54026ae42b6ca33d9faf9, entries=150, sequenceid=83, filesize=11.6 K 2023-04-07 09:54:44,496 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/13454b85e08d4f47a59356c4880ab546 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/13454b85e08d4f47a59356c4880ab546 2023-04-07 09:54:44,503 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/13454b85e08d4f47a59356c4880ab546, entries=150, sequenceid=83, filesize=11.6 K 2023-04-07 09:54:44,504 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 243bf66836bfdaa40ee21a1c79eaec90 in 1266ms, sequenceid=83, compaction requested=false 2023-04-07 09:54:44,505 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:44,505 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:44,505 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:44,505 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:44,505 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:44,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:44,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:44,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee1e3ab[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:44,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:44,540 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,542 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:44,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:44,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:44,543 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,543 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:44,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:44,545 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-07 09:54:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680861283362, max timestamp=1680861284543Num uniques 859; 2023-04-07 09:54:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861283362, max timestamp=1680861284543Num uniques 859; 2023-04-07 09:54:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:54:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680861283362, max timestamp=1680861284543Num uniques 859; 2023-04-07 09:54:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:44,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:44,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:44,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:44,549 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:44,550 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:44,550 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:44,550 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:44,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:44,550 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:44,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:44,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:44,558 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,558 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:44,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,558 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/61b77bdda583422fb060940c34683ca3 2023-04-07 09:54:44,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:44,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:44,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:44,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:44,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:44,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:44,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:44,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,573 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:44,573 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:44,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861284571, max timestamp=1680861284583Num uniques 859; 2023-04-07 09:54:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861284571, max timestamp=1680861284583Num uniques 859; 2023-04-07 09:54:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861284571, max timestamp=1680861284583Num uniques 859; 2023-04-07 09:54:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:44,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/7168cf55c50d48e88edfcc0b9bf81e82 2023-04-07 09:54:44,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.26 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/3e29587626bb4850a11365d2bb335d08 2023-04-07 09:54:44,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/61b77bdda583422fb060940c34683ca3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/61b77bdda583422fb060940c34683ca3 2023-04-07 09:54:44,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/61b77bdda583422fb060940c34683ca3, entries=150, sequenceid=107, filesize=11.6 K 2023-04-07 09:54:44,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/7168cf55c50d48e88edfcc0b9bf81e82 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7168cf55c50d48e88edfcc0b9bf81e82 2023-04-07 09:54:44,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7168cf55c50d48e88edfcc0b9bf81e82, entries=150, sequenceid=107, filesize=11.6 K 2023-04-07 09:54:44,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/3e29587626bb4850a11365d2bb335d08 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3e29587626bb4850a11365d2bb335d08 2023-04-07 09:54:44,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3e29587626bb4850a11365d2bb335d08, entries=150, sequenceid=107, filesize=11.6 K 2023-04-07 09:54:44,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~51.04 KB/52266, heapSize ~135.20 KB/138440, currentSize=120.76 KB/123660 for 243bf66836bfdaa40ee21a1c79eaec90 in 91ms, sequenceid=107, compaction requested=true 2023-04-07 09:54:44,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:44,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:44,637 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:44,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:44,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:54:44,638 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:44,640 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:44,640 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:44,640 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:44,640 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/082da09d8fe344d189e06a0d09beacce, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3dcdc94da10a4e4dad3223e1791e50fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/61b77bdda583422fb060940c34683ca3] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.0 K 2023-04-07 09:54:44,641 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 082da09d8fe344d189e06a0d09beacce, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680861282729 2023-04-07 09:54:44,641 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3dcdc94da10a4e4dad3223e1791e50fd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680861282741 2023-04-07 09:54:44,641 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 61b77bdda583422fb060940c34683ca3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680861284543 2023-04-07 09:54:44,642 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:44,642 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:44,642 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:44,642 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e988fe4ab3a74b19b65d3daaae5526cd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4319a3905c54026ae42b6ca33d9faf9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7168cf55c50d48e88edfcc0b9bf81e82] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.0 K 2023-04-07 09:54:44,643 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e988fe4ab3a74b19b65d3daaae5526cd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680861282729 2023-04-07 09:54:44,643 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a4319a3905c54026ae42b6ca33d9faf9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680861282741 2023-04-07 09:54:44,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:44,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=127.47 KB heapSize=365.53 KB 2023-04-07 09:54:44,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:44,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:44,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:44,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:44,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:44,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:44,646 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7168cf55c50d48e88edfcc0b9bf81e82, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680861284543 2023-04-07 09:54:44,657 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:44,658 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:44,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861344656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861344658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861344658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861344659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861344660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,673 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:44,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:44,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:44,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:44,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:44,712 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:44,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:44,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:44,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:44,713 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,713 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:44,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:44,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:44,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:44,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/07f3b8bf63b04f1ea0c73c4776d94486 2023-04-07 09:54:44,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:44,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:44,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:44,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:44,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:44,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:44,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:44,748 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:44,748 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:44,748 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:44,748 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:44,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:44,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:44,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:44,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:44,751 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6984de[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:44,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:44,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,755 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,755 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:44,755 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:44,755 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:44,756 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,756 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:44,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/09e79d7107ce4c33a89e01031baf9b66 2023-04-07 09:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:44,760 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861344764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:44,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,767 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:44,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:44,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:44,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:44,768 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861344765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861344766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861344766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:44,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,773 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861344769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:44,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:44,776 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:44,776 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:44,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:44,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:44,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:44,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:44,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:44,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:44,877 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:44,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:44,881 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:44,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:44,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:44,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:44,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:44,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:44,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:44,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:44,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:44,885 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:44,885 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:44,885 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:44,885 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:44,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:44,889 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:44,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bcee917[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:44,890 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:44,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:44,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:44,895 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:44,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:44,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:44,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:44,896 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:44,897 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:44,898 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,898 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:44,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:44,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:44,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:44,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:44,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:44,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:44,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:44,904 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:44,904 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:44,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861344968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861344970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861344970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861344971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:44,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:44,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861344975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,004 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:45,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:45,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:45,007 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:45,007 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,007 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:45,007 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:45,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,009 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:45,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:45,010 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,010 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:45,011 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:45,011 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:45,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:45,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@144750ed[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:45,014 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:45,014 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,015 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,015 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,015 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:45,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:45,024 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:45,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:45,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:45,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:45,024 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:45,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,039 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:45,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:45,042 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,042 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,042 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,043 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,140 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/7249f3ab34414c2b924c4898b11624fe as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7249f3ab34414c2b924c4898b11624fe 2023-04-07 09:54:45,143 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:45,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:45,146 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into 7249f3ab34414c2b924c4898b11624fe(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:45,146 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,146 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861284637; duration=0sec 2023-04-07 09:54:45,146 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:45,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:45,146 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:45,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:45,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:45,148 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:45,148 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,148 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:45,148 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:45,149 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:45,149 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:45,149 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,149 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,149 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,149 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3970929a211a489bb8e0d603f394846d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/13454b85e08d4f47a59356c4880ab546, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3e29587626bb4850a11365d2bb335d08] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.0 K 2023-04-07 09:54:45,149 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3970929a211a489bb8e0d603f394846d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680861282729 2023-04-07 09:54:45,150 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 13454b85e08d4f47a59356c4880ab546, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680861282741 2023-04-07 09:54:45,150 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3e29587626bb4850a11365d2bb335d08, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680861284543 2023-04-07 09:54:45,150 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/14964efc15e948e9b02b4211709384b5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/14964efc15e948e9b02b4211709384b5 2023-04-07 09:54:45,155 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,156 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,156 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:45,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:45,157 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 14964efc15e948e9b02b4211709384b5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:45,157 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,157 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861284637; duration=0sec 2023-04-07 09:54:45,157 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:45,158 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#257 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:45,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:45,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:45,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:45,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:45,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:45,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:45,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ca0575[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:45,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:45,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:45,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:45,167 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/0ac1e066ab0841849e07893b244113c1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0ac1e066ab0841849e07893b244113c1 2023-04-07 09:54:45,167 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:45,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:45,170 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:45,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:45,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:45,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:45,170 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:45,172 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 0ac1e066ab0841849e07893b244113c1(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:45,172 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,172 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=13, startTime=1680861284638; duration=0sec 2023-04-07 09:54:45,172 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:45,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,176 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:45,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:45,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,179 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:45,179 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:45,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/64655276e60a4523951c2b12c4e17ff1 2023-04-07 09:54:45,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/07f3b8bf63b04f1ea0c73c4776d94486 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/07f3b8bf63b04f1ea0c73c4776d94486 2023-04-07 09:54:45,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/07f3b8bf63b04f1ea0c73c4776d94486, entries=150, sequenceid=145, filesize=11.8 K 2023-04-07 09:54:45,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/09e79d7107ce4c33a89e01031baf9b66 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/09e79d7107ce4c33a89e01031baf9b66 2023-04-07 09:54:45,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/09e79d7107ce4c33a89e01031baf9b66, entries=150, sequenceid=145, filesize=11.8 K 2023-04-07 09:54:45,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/64655276e60a4523951c2b12c4e17ff1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/64655276e60a4523951c2b12c4e17ff1 2023-04-07 09:54:45,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/64655276e60a4523951c2b12c4e17ff1, entries=150, sequenceid=145, filesize=11.8 K 2023-04-07 09:54:45,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=46.96 KB/48090 for 243bf66836bfdaa40ee21a1c79eaec90 in 597ms, sequenceid=145, compaction requested=false 2023-04-07 09:54:45,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:45,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:54:45,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:45,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:45,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:45,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,279 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:45,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:45,285 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:45,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:45,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:45,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,287 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:45,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:45,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:45,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:45,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:45,293 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:45,297 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,300 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/7f82b399285c4f24bdd4399baed95cba 2023-04-07 09:54:45,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:45,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42222266[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:45,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:45,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,311 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:45,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:45,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:45,312 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,312 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:45,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:45,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:45,314 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861345301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:45,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:45,319 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:45,319 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:45,319 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:45,319 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:45,319 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:45,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861345317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861345317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,328 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:45,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,328 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:45,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861345328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,328 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:45,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:45,328 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861345303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/27334d48face42cfa2ae4d503b26fe19 2023-04-07 09:54:45,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861345418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861345425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861345425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,429 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:45,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:45,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861345429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:45,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:45,435 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:45,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:45,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:45,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861345433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,442 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:45,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:45,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:45,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:45,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:45,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:45,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:45,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e28fd64[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:45,448 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:45,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,450 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:45,450 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:45,450 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,450 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:45,451 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:45,451 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,451 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:45,451 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:45,454 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:45,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:45,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:45,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:45,454 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:45,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:45,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:45,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:45,463 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:45,463 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:45,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,563 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:45,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:45,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:45,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:45,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:45,566 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:45,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,567 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:45,567 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:45,567 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,567 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,569 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,569 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:45,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:45,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:45,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:45,570 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,570 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:45,570 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:45,571 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:45,575 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,575 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:45,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60461c74[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:45,577 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:45,577 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,578 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:45,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:45,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:45,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:45,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:45,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:45,582 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:45,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:45,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:45,583 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:45,583 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:45,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:45,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:45,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,592 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:45,592 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:45,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861345621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861345629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861345629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861345633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861345637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,693 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:45,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:45,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:45,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:45,696 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:45,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:45,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:45,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,698 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,698 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,698 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:45,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:45,699 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,699 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:45,699 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:45,699 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:45,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55125ca4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:45,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:45,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,703 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,704 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:45,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:45,704 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:45,704 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:45,706 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:45,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:45,709 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:45,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:45,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:45,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:45,709 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:45,710 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,710 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,710 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:45,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:45,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,714 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,714 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:45,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:45,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/e93aa737083246298432ff2691fb5d2c 2023-04-07 09:54:45,789 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/7f82b399285c4f24bdd4399baed95cba as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/7f82b399285c4f24bdd4399baed95cba 2023-04-07 09:54:45,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/7f82b399285c4f24bdd4399baed95cba, entries=200, sequenceid=162, filesize=14.1 K 2023-04-07 09:54:45,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/27334d48face42cfa2ae4d503b26fe19 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/27334d48face42cfa2ae4d503b26fe19 2023-04-07 09:54:45,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/27334d48face42cfa2ae4d503b26fe19, entries=150, sequenceid=162, filesize=11.8 K 2023-04-07 09:54:45,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/e93aa737083246298432ff2691fb5d2c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e93aa737083246298432ff2691fb5d2c 2023-04-07 09:54:45,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e93aa737083246298432ff2691fb5d2c, entries=150, sequenceid=162, filesize=11.8 K 2023-04-07 09:54:45,804 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 243bf66836bfdaa40ee21a1c79eaec90 in 527ms, sequenceid=162, compaction requested=true 2023-04-07 09:54:45,804 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:45,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:45,804 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:45,804 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:45,804 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:45,805 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38588 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:45,805 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:45,805 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,805 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/14964efc15e948e9b02b4211709384b5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/07f3b8bf63b04f1ea0c73c4776d94486, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/7f82b399285c4f24bdd4399baed95cba] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=37.7 K 2023-04-07 09:54:45,805 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:45,806 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:45,806 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,806 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7249f3ab34414c2b924c4898b11624fe, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/09e79d7107ce4c33a89e01031baf9b66, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/27334d48face42cfa2ae4d503b26fe19] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.3 K 2023-04-07 09:54:45,806 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 14964efc15e948e9b02b4211709384b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680861284543 2023-04-07 09:54:45,806 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7249f3ab34414c2b924c4898b11624fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680861284543 2023-04-07 09:54:45,806 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 07f3b8bf63b04f1ea0c73c4776d94486, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680861284639 2023-04-07 09:54:45,806 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 09e79d7107ce4c33a89e01031baf9b66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680861284639 2023-04-07 09:54:45,807 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 7f82b399285c4f24bdd4399baed95cba, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861284657 2023-04-07 09:54:45,807 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 27334d48face42cfa2ae4d503b26fe19, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861284657 2023-04-07 09:54:45,815 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#262 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:45,815 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:45,817 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:45,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:45,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:45,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:45,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:45,822 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:45,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:45,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:45,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,823 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,824 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,824 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,824 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,824 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:45,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:45,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:45,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:45,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:45,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:45,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:45,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:45,829 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:45,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,833 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/2fd6953f99f24ff783b4305e853e7152 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2fd6953f99f24ff783b4305e853e7152 2023-04-07 09:54:45,837 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into 2fd6953f99f24ff783b4305e853e7152(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:45,837 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,837 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861285804; duration=0sec 2023-04-07 09:54:45,837 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:45,837 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:45,841 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:45,841 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:45,841 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,841 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0ac1e066ab0841849e07893b244113c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/64655276e60a4523951c2b12c4e17ff1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e93aa737083246298432ff2691fb5d2c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.3 K 2023-04-07 09:54:45,841 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0ac1e066ab0841849e07893b244113c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680861284543 2023-04-07 09:54:45,842 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 64655276e60a4523951c2b12c4e17ff1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680861284639 2023-04-07 09:54:45,842 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e93aa737083246298432ff2691fb5d2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861284657 2023-04-07 09:54:45,845 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef 2023-04-07 09:54:45,849 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:45,867 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/642572c56a7e4b419c9b401963158e87 2023-04-07 09:54:45,869 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/b327d91d28644c95ae022e1f1d113d96 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/b327d91d28644c95ae022e1f1d113d96 2023-04-07 09:54:45,873 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into b327d91d28644c95ae022e1f1d113d96(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:45,873 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,873 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=13, startTime=1680861285804; duration=0sec 2023-04-07 09:54:45,874 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:45,884 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/27977751b2e1447bab5ea6cf89873749 2023-04-07 09:54:45,888 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef 2023-04-07 09:54:45,891 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef, entries=150, sequenceid=182, filesize=11.8 K 2023-04-07 09:54:45,892 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/642572c56a7e4b419c9b401963158e87 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/642572c56a7e4b419c9b401963158e87 2023-04-07 09:54:45,896 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/642572c56a7e4b419c9b401963158e87, entries=150, sequenceid=182, filesize=11.8 K 2023-04-07 09:54:45,896 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/27977751b2e1447bab5ea6cf89873749 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/27977751b2e1447bab5ea6cf89873749 2023-04-07 09:54:45,900 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/27977751b2e1447bab5ea6cf89873749, entries=150, sequenceid=182, filesize=11.8 K 2023-04-07 09:54:45,900 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 243bf66836bfdaa40ee21a1c79eaec90 in 71ms, sequenceid=182, compaction requested=false 2023-04-07 09:54:45,900 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:45,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:45,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:45,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6706572[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:45,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:45,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,907 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:45,908 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,908 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:45,909 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:45,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:45,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:45,912 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:45,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:45,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:45,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:45,912 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:45,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:45,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:45,919 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:45,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:45,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:45,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:45,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:45,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:45,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:45,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:45,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:45,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:45,920 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:45,920 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:45,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:45,935 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:54:45,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:45,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:45,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:45,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:45,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861285948, max timestamp=1680861285951Num uniques 859; 2023-04-07 09:54:45,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861345949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:45,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861285948, max timestamp=1680861285951Num uniques 859; 2023-04-07 09:54:45,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:45,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861345960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861345961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861345962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:45,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861345963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,020 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:46,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:46,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:46,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:46,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:46,024 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:46,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:46,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:46,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:46,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:46,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:46,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f3a6b05[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:46,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,031 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:46,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:46,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:46,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:46,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:46,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:46,040 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:46,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:46,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:46,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:46,040 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:46,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,043 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,044 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:46,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:46,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:46,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,047 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:46,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,047 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:46,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861346053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861346064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861346066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861346067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861346067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,147 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:46,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:46,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:46,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:46,151 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:46,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:46,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:46,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,152 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:46,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:46,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:46,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:46,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:46,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:46,158 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:46,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fcf732a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:46,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:46,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:46,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:46,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:46,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:46,168 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:46,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:46,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:46,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:46,168 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:46,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:46,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:46,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:46,179 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:46,179 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:46,232 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/2c4e1ca273c842b5b530d7ce418e75fa as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/2c4e1ca273c842b5b530d7ce418e75fa 2023-04-07 09:54:46,236 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 2c4e1ca273c842b5b530d7ce418e75fa(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:46,236 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:46,236 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861285804; duration=0sec 2023-04-07 09:54:46,236 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:46,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861346257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861346268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861346270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861346270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861346271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,280 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:46,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:46,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:46,283 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,285 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:46,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,286 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:46,292 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,292 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:46,292 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:46,292 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:46,294 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:46,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187ff30b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:46,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:46,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,297 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:46,298 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:46,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,300 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:46,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:46,304 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:46,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:46,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:46,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:46,304 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:46,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,313 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:46,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,313 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:46,313 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:46,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/ca95465ff02e4c55a8afd5b22d295f53 2023-04-07 09:54:46,413 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:46,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:46,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:46,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:46,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:46,416 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:46,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:46,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:46,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,418 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,419 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,419 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:46,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:46,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:46,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:46,420 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,420 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:46,420 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:46,420 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:46,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:46,422 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,423 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,423 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:46,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d215e83[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:46,424 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:46,424 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:46,425 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,425 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:46,427 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:46,429 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:46,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:46,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:46,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:46,429 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:46,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,435 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:46,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,443 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:46,443 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:46,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,544 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:46,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:46,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:46,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:46,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:46,547 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:46,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:46,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:46,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:46,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:46,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:46,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:46,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:46,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:46,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@419f7c61[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:46,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:46,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:46,556 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,556 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:46,558 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:46,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:46,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:46,560 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:46,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:46,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:46,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:46,560 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:46,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861346561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:46,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:46,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,567 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:46,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,567 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:46,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861346570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861346573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861346574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:46,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861346574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,667 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:46,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:46,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:46,671 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:46,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:46,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:46,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,674 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,674 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:46,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:46,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:46,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:46,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:46,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:46,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:46,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:46,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:46,676 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:46,677 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:46,678 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,678 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,678 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,678 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:46,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ed115b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:46,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:46,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:46,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:46,681 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:46,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:46,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:46,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:46,695 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:46,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:46,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:46,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:46,695 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:46,698 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,698 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,698 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:46,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:46,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:46,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:46,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:46,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:46,708 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:46,708 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:46,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:46,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:46,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/bcc0559fb2d2477e8a2b88cdbcd33efb 2023-04-07 09:54:46,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/09249f30e3d84fdabf07b5a295c8313a 2023-04-07 09:54:46,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/ca95465ff02e4c55a8afd5b22d295f53 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ca95465ff02e4c55a8afd5b22d295f53 2023-04-07 09:54:46,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ca95465ff02e4c55a8afd5b22d295f53, entries=150, sequenceid=195, filesize=11.8 K 2023-04-07 09:54:46,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/bcc0559fb2d2477e8a2b88cdbcd33efb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/bcc0559fb2d2477e8a2b88cdbcd33efb 2023-04-07 09:54:46,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/bcc0559fb2d2477e8a2b88cdbcd33efb, entries=150, sequenceid=195, filesize=11.8 K 2023-04-07 09:54:46,781 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/09249f30e3d84fdabf07b5a295c8313a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/09249f30e3d84fdabf07b5a295c8313a 2023-04-07 09:54:46,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/09249f30e3d84fdabf07b5a295c8313a, entries=150, sequenceid=195, filesize=11.8 K 2023-04-07 09:54:46,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 243bf66836bfdaa40ee21a1c79eaec90 in 850ms, sequenceid=195, compaction requested=true 2023-04-07 09:54:46,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:46,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:46,785 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:46,785 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:46,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:46,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:46,786 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:46,786 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:46,786 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:46,786 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:46,786 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,786 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,786 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/2c4e1ca273c842b5b530d7ce418e75fa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ca95465ff02e4c55a8afd5b22d295f53] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.6 K 2023-04-07 09:54:46,786 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2fd6953f99f24ff783b4305e853e7152, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/642572c56a7e4b419c9b401963158e87, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/bcc0559fb2d2477e8a2b88cdbcd33efb] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.6 K 2023-04-07 09:54:46,787 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 2c4e1ca273c842b5b530d7ce418e75fa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861284657 2023-04-07 09:54:46,787 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2fd6953f99f24ff783b4305e853e7152, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861284657 2023-04-07 09:54:46,787 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 9d8cbecbf3ad4b08be3a1ffe9f1f08ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680861285294 2023-04-07 09:54:46,787 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 642572c56a7e4b419c9b401963158e87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680861285294 2023-04-07 09:54:46,787 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ca95465ff02e4c55a8afd5b22d295f53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680861285932 2023-04-07 09:54:46,787 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting bcc0559fb2d2477e8a2b88cdbcd33efb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680861285932 2023-04-07 09:54:46,793 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:46,793 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#271 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:46,808 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:46,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:46,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:46,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:46,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:46,811 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:46,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:46,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:46,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,812 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,813 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:46,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:46,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:46,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:46,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:46,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:46,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:46,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:46,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:46,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:46,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:46,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:46,814 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:46,814 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:46,814 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=125.23 KB heapSize=363.39 KB 2023-04-07 09:54:46,815 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:46,815 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:46,815 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:46,815 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:46,815 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:46,815 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:46,821 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/f7c0bb1354b04d78aa826b434e3d49f9 2023-04-07 09:54:47,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:47,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:47,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861347079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861347079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861347081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861347082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861347083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861347186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861347187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861347187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,202 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3e11c230cf484240934cc6393b72e547 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3e11c230cf484240934cc6393b72e547 2023-04-07 09:54:47,205 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/94cfc709e6f94b629efe25422a1dee9b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/94cfc709e6f94b629efe25422a1dee9b 2023-04-07 09:54:47,207 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 3e11c230cf484240934cc6393b72e547(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:47,207 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:47,207 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861286785; duration=0sec 2023-04-07 09:54:47,207 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:47,207 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:47,209 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:47,209 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:47,209 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,209 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/b327d91d28644c95ae022e1f1d113d96, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/27977751b2e1447bab5ea6cf89873749, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/09249f30e3d84fdabf07b5a295c8313a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.6 K 2023-04-07 09:54:47,211 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b327d91d28644c95ae022e1f1d113d96, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861284657 2023-04-07 09:54:47,212 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 27977751b2e1447bab5ea6cf89873749, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680861285294 2023-04-07 09:54:47,212 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 09249f30e3d84fdabf07b5a295c8313a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680861285932 2023-04-07 09:54:47,218 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into 94cfc709e6f94b629efe25422a1dee9b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:47,218 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:47,218 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861286785; duration=0sec 2023-04-07 09:54:47,219 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:47,221 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:47,234 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/fe1ab0a1e07448c2a8adc7afd21a9977 2023-04-07 09:54:47,280 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/60c9c3b9e0df40eeb1a181e41038e4de as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/60c9c3b9e0df40eeb1a181e41038e4de 2023-04-07 09:54:47,284 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 60c9c3b9e0df40eeb1a181e41038e4de(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:47,284 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:47,284 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=13, startTime=1680861286785; duration=0sec 2023-04-07 09:54:47,285 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:47,291 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/8349bcd2629f4c3c85a424419a26f8eb 2023-04-07 09:54:47,296 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/f7c0bb1354b04d78aa826b434e3d49f9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/f7c0bb1354b04d78aa826b434e3d49f9 2023-04-07 09:54:47,300 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/f7c0bb1354b04d78aa826b434e3d49f9, entries=150, sequenceid=227, filesize=11.8 K 2023-04-07 09:54:47,301 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/fe1ab0a1e07448c2a8adc7afd21a9977 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fe1ab0a1e07448c2a8adc7afd21a9977 2023-04-07 09:54:47,305 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fe1ab0a1e07448c2a8adc7afd21a9977, entries=150, sequenceid=227, filesize=11.8 K 2023-04-07 09:54:47,305 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-07 09:54:47,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/8349bcd2629f4c3c85a424419a26f8eb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/8349bcd2629f4c3c85a424419a26f8eb 2023-04-07 09:54:47,309 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/8349bcd2629f4c3c85a424419a26f8eb, entries=150, sequenceid=227, filesize=11.8 K 2023-04-07 09:54:47,310 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~363.34 KB/372064, currentSize=53.67 KB/54960 for 243bf66836bfdaa40ee21a1c79eaec90 in 496ms, sequenceid=227, compaction requested=false 2023-04-07 09:54:47,310 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:47,310 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:47,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:47,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:47,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:47,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed4146[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:47,315 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:47,315 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,318 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:47,318 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,318 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:47,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:47,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:47,323 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:47,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:47,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:47,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:47,323 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:47,324 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,324 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,324 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,326 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:47,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:47,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,333 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:47,334 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:47,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:47,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:47,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:54:47,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:47,394 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:47,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:47,394 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:47,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:47,394 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:47,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3c031fb0ea654aa995c678a17f770f89 2023-04-07 09:54:47,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861347415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/d42b7053a28948559f00e7b286757683 2023-04-07 09:54:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861347417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861347417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,434 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:47,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:47,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:47,452 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:47,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:47,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:47,453 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,453 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:47,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:47,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:47,455 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:47,455 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,455 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:47,455 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:47,455 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:47,455 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,456 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:47,456 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:47,456 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:47,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:47,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:47,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:47,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152fd2de[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:47,459 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:47,459 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,460 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:47,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:47,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:47,464 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:47,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:47,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:47,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:47,465 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:47,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:47,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:47,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,470 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:47,471 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:47,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861347519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861347522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861347522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,571 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:47,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:47,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:47,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:47,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:47,574 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:47,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:47,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:47,575 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,575 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:47,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:47,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,577 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:47,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:47,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:47,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:47,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:47,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,581 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,581 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f69d86e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:47,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:47,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:47,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:47,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:47,583 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,583 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:47,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:47,587 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:47,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:47,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:47,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:47,587 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:47,588 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,588 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,588 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,590 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,593 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:47,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:47,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:47,594 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:47,594 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:47,694 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:47,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:47,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:47,697 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:47,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:47,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:47,701 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,702 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:47,702 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:47,702 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:47,703 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,704 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22837311[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:47,705 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:47,705 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,706 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:47,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:47,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:47,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:47,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:47,711 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:47,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:47,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:47,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:47,711 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:47,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,713 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:47,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:47,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:47,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,717 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:47,717 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:47,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861347722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861347726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:47,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861347726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,817 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:47,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:47,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:47,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:47,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:47,821 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:47,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:47,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:47,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:47,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:47,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:47,827 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:47,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:47,829 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:47,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0cf176[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:47,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:47,831 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:47,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:47,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:47,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:47,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:47,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:47,838 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:47,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:47,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:47,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:47,838 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:47,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:47,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:47,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,845 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,845 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:47,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:47,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:47,845 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:47,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:47,845 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:47,845 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:47,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:47,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/e27a494f78774d1a9836352b57c60413 2023-04-07 09:54:47,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3c031fb0ea654aa995c678a17f770f89 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c031fb0ea654aa995c678a17f770f89 2023-04-07 09:54:47,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c031fb0ea654aa995c678a17f770f89, entries=150, sequenceid=243, filesize=11.8 K 2023-04-07 09:54:47,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/d42b7053a28948559f00e7b286757683 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d42b7053a28948559f00e7b286757683 2023-04-07 09:54:47,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d42b7053a28948559f00e7b286757683, entries=150, sequenceid=243, filesize=11.8 K 2023-04-07 09:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/e27a494f78774d1a9836352b57c60413 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e27a494f78774d1a9836352b57c60413 2023-04-07 09:54:47,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e27a494f78774d1a9836352b57c60413, entries=150, sequenceid=243, filesize=11.8 K 2023-04-07 09:54:47,876 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 243bf66836bfdaa40ee21a1c79eaec90 in 484ms, sequenceid=243, compaction requested=true 2023-04-07 09:54:47,876 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:47,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:47,876 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:47,876 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:47,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:47,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:47,877 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:47,877 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:47,877 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,877 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3e11c230cf484240934cc6393b72e547, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/f7c0bb1354b04d78aa826b434e3d49f9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c031fb0ea654aa995c678a17f770f89] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.7 K 2023-04-07 09:54:47,880 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:47,880 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:47,880 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,880 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/94cfc709e6f94b629efe25422a1dee9b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fe1ab0a1e07448c2a8adc7afd21a9977, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d42b7053a28948559f00e7b286757683] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.7 K 2023-04-07 09:54:47,881 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3e11c230cf484240934cc6393b72e547, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680861285932 2023-04-07 09:54:47,881 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 94cfc709e6f94b629efe25422a1dee9b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680861285932 2023-04-07 09:54:47,882 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting f7c0bb1354b04d78aa826b434e3d49f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680861285953 2023-04-07 09:54:47,882 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fe1ab0a1e07448c2a8adc7afd21a9977, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680861285953 2023-04-07 09:54:47,882 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3c031fb0ea654aa995c678a17f770f89, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680861287081 2023-04-07 09:54:47,882 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d42b7053a28948559f00e7b286757683, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680861287081 2023-04-07 09:54:47,897 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:47,897 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:47,918 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/bf55f472792e4d21a2e0dc935b10aa1d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/bf55f472792e4d21a2e0dc935b10aa1d 2023-04-07 09:54:47,923 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into bf55f472792e4d21a2e0dc935b10aa1d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:47,923 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:47,923 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861287876; duration=0sec 2023-04-07 09:54:47,924 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:47,924 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:47,925 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:47,925 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:47,925 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,925 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/60c9c3b9e0df40eeb1a181e41038e4de, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/8349bcd2629f4c3c85a424419a26f8eb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e27a494f78774d1a9836352b57c60413] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=35.7 K 2023-04-07 09:54:47,925 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 60c9c3b9e0df40eeb1a181e41038e4de, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1680861285932 2023-04-07 09:54:47,926 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8349bcd2629f4c3c85a424419a26f8eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680861285953 2023-04-07 09:54:47,926 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e27a494f78774d1a9836352b57c60413, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680861287081 2023-04-07 09:54:47,931 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:47,945 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:47,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:47,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:47,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:47,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:47,949 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:47,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:47,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:47,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,950 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:47,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:47,951 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,951 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:47,951 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:47,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:47,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:47,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:47,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:47,953 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:47,953 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:47,953 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:54:47,954 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:47,954 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:47,954 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:47,954 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:47,954 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:47,954 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:47,954 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/ef260c40f49945ebb1b72717492892a9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/ef260c40f49945ebb1b72717492892a9 2023-04-07 09:54:47,960 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into ef260c40f49945ebb1b72717492892a9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:47,960 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:47,960 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=13, startTime=1680861287876; duration=0sec 2023-04-07 09:54:47,960 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:47,970 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/98fe87129f6b43d0b36a57a7c190f697 2023-04-07 09:54:48,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:48,027 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:48,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861348037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861348038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861348039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861348086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861348086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861348141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861348141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861348141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,316 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/c8f764706144414bb3fe0da7073d3ed3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/c8f764706144414bb3fe0da7073d3ed3 2023-04-07 09:54:48,320 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into c8f764706144414bb3fe0da7073d3ed3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:48,320 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:48,320 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861287876; duration=0sec 2023-04-07 09:54:48,320 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:48,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861348344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861348344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861348344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,406 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/313a0b48679e417e9bdad89f6423b960 2023-04-07 09:54:48,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861348646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861348647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861348647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,845 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/10adf5eb1e0c414ea3d26bc933ee82ea 2023-04-07 09:54:48,850 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/98fe87129f6b43d0b36a57a7c190f697 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/98fe87129f6b43d0b36a57a7c190f697 2023-04-07 09:54:48,853 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/98fe87129f6b43d0b36a57a7c190f697, entries=150, sequenceid=263, filesize=11.9 K 2023-04-07 09:54:48,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/313a0b48679e417e9bdad89f6423b960 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/313a0b48679e417e9bdad89f6423b960 2023-04-07 09:54:48,857 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/313a0b48679e417e9bdad89f6423b960, entries=150, sequenceid=263, filesize=11.9 K 2023-04-07 09:54:48,858 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/10adf5eb1e0c414ea3d26bc933ee82ea as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/10adf5eb1e0c414ea3d26bc933ee82ea 2023-04-07 09:54:48,861 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/10adf5eb1e0c414ea3d26bc933ee82ea, entries=150, sequenceid=263, filesize=11.9 K 2023-04-07 09:54:48,862 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 243bf66836bfdaa40ee21a1c79eaec90 in 908ms, sequenceid=263, compaction requested=false 2023-04-07 09:54:48,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:48,862 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:48,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:48,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:48,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21de73cb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:48,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:48,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,868 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:48,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:48,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:48,870 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:48,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:48,873 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:48,873 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:48,873 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:48,873 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:48,873 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:48,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:48,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:48,875 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:48,877 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:48,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:48,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:48,879 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:48,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:48,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:48,879 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:48,879 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:48,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,979 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:48,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:48,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:48,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:48,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:48,984 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:48,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:48,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:48,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:48,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:48,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:48,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:48,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:48,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:48,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:48,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:48,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:48,989 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:49,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:49,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:49,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861349167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861349169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861349170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861349271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1680861349273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861349273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,402 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3c4363b38abd4653a4f7d9675c33973f 2023-04-07 09:54:49,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861349474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861349477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861349477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48332 deadline: 1680861349778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861349780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:49,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861349780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:49,811 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/e138b8c8802341f0ab7ed33cf37b307d 2023-04-07 09:54:50,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861350099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,101 DEBUG [Thread-1836] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:54:50,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861350104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,107 DEBUG [Thread-1832] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:54:50,221 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/4a527a21f4aa4f678e11ad224643c9c4 2023-04-07 09:54:50,226 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/3c4363b38abd4653a4f7d9675c33973f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c4363b38abd4653a4f7d9675c33973f 2023-04-07 09:54:50,230 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c4363b38abd4653a4f7d9675c33973f, entries=150, sequenceid=278, filesize=11.9 K 2023-04-07 09:54:50,230 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/e138b8c8802341f0ab7ed33cf37b307d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e138b8c8802341f0ab7ed33cf37b307d 2023-04-07 09:54:50,234 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e138b8c8802341f0ab7ed33cf37b307d, entries=150, sequenceid=278, filesize=11.9 K 2023-04-07 09:54:50,235 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/4a527a21f4aa4f678e11ad224643c9c4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/4a527a21f4aa4f678e11ad224643c9c4 2023-04-07 09:54:50,238 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/4a527a21f4aa4f678e11ad224643c9c4, entries=150, sequenceid=278, filesize=11.9 K 2023-04-07 09:54:50,238 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 243bf66836bfdaa40ee21a1c79eaec90 in 1249ms, sequenceid=278, compaction requested=true 2023-04-07 09:54:50,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:50,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:50,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:50,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:50,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:50,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40156536[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:50,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:50,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,244 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:50,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:50,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,245 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,246 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,246 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:50,248 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:50,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:50,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:50,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:50,248 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:50,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,253 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,256 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:50,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:50,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,256 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:50,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,256 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:50,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:50,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:54:50,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:50,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861350295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861350296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861350297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,356 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:50,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:50,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:50,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:50,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:50,361 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:50,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:50,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:50,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,363 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:50,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:50,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:50,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:50,379 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,379 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79b79dde[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:50,382 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:50,382 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,383 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:50,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:50,383 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:50,384 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,384 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:50,386 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:50,388 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:50,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:50,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:50,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:50,388 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:50,392 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,392 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,392 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:50,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:50,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:50,396 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:50,396 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:50,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861350398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48318 deadline: 1680861350399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861350399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,496 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:50,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:50,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:50,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:50,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:50,500 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:50,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:50,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:50,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,501 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,502 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,502 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:50,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:50,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:50,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:50,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:50,503 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:50,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,505 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:50,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@549d54df[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:50,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:50,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,509 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,509 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:50,509 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:50,509 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:50,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:50,510 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:50,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:50,513 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:50,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:50,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:50,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:50,513 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:50,514 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,514 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,514 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,516 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:50,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:50,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:50,518 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,519 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,519 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,519 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:50,519 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861350602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861350603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861350603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,619 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:50,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:50,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:50,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:50,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:50,622 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:50,622 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,622 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:50,622 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:50,623 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,623 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,625 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:50,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:50,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:50,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:50,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:50,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:50,629 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:50,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f25ac[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:50,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:50,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,632 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:50,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:50,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:50,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:50,633 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:50,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:50,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:50,636 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:50,636 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:50,636 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:50,636 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:50,636 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:50,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,642 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:50,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:50,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,646 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:50,646 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:50,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/83bc7e239c5747b7bace99256c5e6351 2023-04-07 09:54:50,746 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:50,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:50,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:50,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:50,749 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:50,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:50,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:50,750 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,750 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,750 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:50,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:50,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:50,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:50,752 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:50,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,754 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,754 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,754 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:50,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd8dc9e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:50,755 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:50,755 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:50,758 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,758 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:50,762 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:50,764 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:50,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:50,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:50,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:50,764 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:50,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,766 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,767 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:50,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:50,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,770 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,770 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:50,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,870 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:50,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:50,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:50,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:50,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:50,874 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:50,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:50,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:50,875 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,875 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:50,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:50,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:50,880 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:50,884 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,884 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:50,884 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:50,884 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:50,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,886 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2844f596[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:50,888 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:50,888 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,889 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,889 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:50,889 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:50,889 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:50,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:50,891 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:50,893 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:50,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:50,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:50,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:50,893 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:50,899 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,899 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,899 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:50,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:50,900 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,902 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:50,902 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:50,902 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:50,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:50,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:50,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:50,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:50,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:50,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:50,903 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:50,903 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:50,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:50,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:50,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861350906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861350907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:50,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:50,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861350907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,003 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:51,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:51,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:51,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:51,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:51,006 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:51,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:51,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:51,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:51,007 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,007 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,008 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:51,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:51,008 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:51,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:51,010 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:51,010 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:51,011 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:51,011 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:51,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:51,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:51,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:51,011 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:51,013 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,013 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:51,015 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:51,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164167ed[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:51,015 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,017 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:51,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:51,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:51,021 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:51,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:51,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:51,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:51,021 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:51,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:51,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:51,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:51,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:51,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:51,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:51,027 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:51,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:51,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:51,026 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:51,027 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:51,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:51,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:51,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/dcf8516bf12d4e55991fdf7bb7a8a49a 2023-04-07 09:54:51,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/818ebbebb437420d9feadca675c179d3 2023-04-07 09:54:51,127 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:51,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:51,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/83bc7e239c5747b7bace99256c5e6351 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/83bc7e239c5747b7bace99256c5e6351 2023-04-07 09:54:51,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:51,130 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:51,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:51,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:51,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:51,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/83bc7e239c5747b7bace99256c5e6351, entries=200, sequenceid=299, filesize=14.3 K 2023-04-07 09:54:51,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/dcf8516bf12d4e55991fdf7bb7a8a49a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/dcf8516bf12d4e55991fdf7bb7a8a49a 2023-04-07 09:54:51,133 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:51,133 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:51,134 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,134 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:51,136 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/dcf8516bf12d4e55991fdf7bb7a8a49a, entries=150, sequenceid=299, filesize=11.9 K 2023-04-07 09:54:51,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/818ebbebb437420d9feadca675c179d3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/818ebbebb437420d9feadca675c179d3 2023-04-07 09:54:51,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/818ebbebb437420d9feadca675c179d3, entries=150, sequenceid=299, filesize=11.9 K 2023-04-07 09:54:51,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 243bf66836bfdaa40ee21a1c79eaec90 in 855ms, sequenceid=299, compaction requested=true 2023-04-07 09:54:51,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:51,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:51,141 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:51,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:51,141 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:51,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:51,142 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51580 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:51,142 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:51,142 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:51,142 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:51,142 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:51,142 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:51,143 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/bf55f472792e4d21a2e0dc935b10aa1d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/98fe87129f6b43d0b36a57a7c190f697, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c4363b38abd4653a4f7d9675c33973f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/83bc7e239c5747b7bace99256c5e6351] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=50.4 K 2023-04-07 09:54:51,143 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/c8f764706144414bb3fe0da7073d3ed3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/313a0b48679e417e9bdad89f6423b960, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e138b8c8802341f0ab7ed33cf37b307d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/dcf8516bf12d4e55991fdf7bb7a8a49a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=48.0 K 2023-04-07 09:54:51,143 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bf55f472792e4d21a2e0dc935b10aa1d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680861287081 2023-04-07 09:54:51,143 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 98fe87129f6b43d0b36a57a7c190f697, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680861287412 2023-04-07 09:54:51,143 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c8f764706144414bb3fe0da7073d3ed3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680861287081 2023-04-07 09:54:51,144 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3c4363b38abd4653a4f7d9675c33973f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680861288036 2023-04-07 09:54:51,144 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 83bc7e239c5747b7bace99256c5e6351, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680861289166 2023-04-07 09:54:51,145 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 313a0b48679e417e9bdad89f6423b960, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680861287412 2023-04-07 09:54:51,145 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e138b8c8802341f0ab7ed33cf37b307d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680861288036 2023-04-07 09:54:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:51,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:51,146 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting dcf8516bf12d4e55991fdf7bb7a8a49a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680861289166 2023-04-07 09:54:51,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:51,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:51,159 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:51,159 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:51,159 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:51,160 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:54:51,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:51,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:51,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:51,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:51,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:51,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:51,161 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#291 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:51,182 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#292 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:51,194 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/ec5ac61e20dc4be58fa32a769e97fb84 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ec5ac61e20dc4be58fa32a769e97fb84 2023-04-07 09:54:51,199 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into ec5ac61e20dc4be58fa32a769e97fb84(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:51,199 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:51,199 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=12, startTime=1680861291141; duration=0sec 2023-04-07 09:54:51,199 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:51,199 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:51,201 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:51,201 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:51,201 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:51,201 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/ef260c40f49945ebb1b72717492892a9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/10adf5eb1e0c414ea3d26bc933ee82ea, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/4a527a21f4aa4f678e11ad224643c9c4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/818ebbebb437420d9feadca675c179d3] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=48.0 K 2023-04-07 09:54:51,201 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ef260c40f49945ebb1b72717492892a9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680861287081 2023-04-07 09:54:51,202 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 10adf5eb1e0c414ea3d26bc933ee82ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680861287412 2023-04-07 09:54:51,202 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4a527a21f4aa4f678e11ad224643c9c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680861288036 2023-04-07 09:54:51,202 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 818ebbebb437420d9feadca675c179d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680861289166 2023-04-07 09:54:51,226 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/2b23888fccdf4ca9b73708b970ae2557 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2b23888fccdf4ca9b73708b970ae2557 2023-04-07 09:54:51,231 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into 2b23888fccdf4ca9b73708b970ae2557(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:51,231 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:51,231 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=12, startTime=1680861291141; duration=0sec 2023-04-07 09:54:51,231 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:51,231 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:51,245 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/15710071aec04ecab21356b5b848bf6d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/15710071aec04ecab21356b5b848bf6d 2023-04-07 09:54:51,253 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 15710071aec04ecab21356b5b848bf6d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:51,254 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:51,254 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=12, startTime=1680861291141; duration=0sec 2023-04-07 09:54:51,254 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:51,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:51,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:51,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861291425, max timestamp=1680861291436Num uniques 859; 2023-04-07 09:54:51,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:51,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861291425, max timestamp=1680861291436Num uniques 859; 2023-04-07 09:54:51,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861291425, max timestamp=1680861291436Num uniques 859; 2023-04-07 09:54:51,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:51,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861351484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861351484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861351484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861351588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861351588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861351589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,634 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/71d55da6139f49c697c7b70789358a60 2023-04-07 09:54:51,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861351792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861351792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:51,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861351793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,047 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/2a069cb6fb5845a5b7b223bccdcf7242 2023-04-07 09:54:52,071 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/90e49a48f9b34933b1144940a7b1097b 2023-04-07 09:54:52,075 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/71d55da6139f49c697c7b70789358a60 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/71d55da6139f49c697c7b70789358a60 2023-04-07 09:54:52,080 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/71d55da6139f49c697c7b70789358a60, entries=100, sequenceid=310, filesize=9.5 K 2023-04-07 09:54:52,080 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/2a069cb6fb5845a5b7b223bccdcf7242 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2a069cb6fb5845a5b7b223bccdcf7242 2023-04-07 09:54:52,084 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2a069cb6fb5845a5b7b223bccdcf7242, entries=100, sequenceid=310, filesize=9.5 K 2023-04-07 09:54:52,085 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/90e49a48f9b34933b1144940a7b1097b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/90e49a48f9b34933b1144940a7b1097b 2023-04-07 09:54:52,088 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/90e49a48f9b34933b1144940a7b1097b, entries=100, sequenceid=310, filesize=9.5 K 2023-04-07 09:54:52,089 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 243bf66836bfdaa40ee21a1c79eaec90 in 929ms, sequenceid=310, compaction requested=false 2023-04-07 09:54:52,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:52,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:52,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691c82cc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:52,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:52,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,097 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:52,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:52,098 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,098 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:52,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861291482, max timestamp=1680861292098Num uniques 1068; 2023-04-07 09:54:52,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:54:52,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861291482, max timestamp=1680861292098Num uniques 1068; 2023-04-07 09:54:52,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861291482, max timestamp=1680861292098Num uniques 1068; 2023-04-07 09:54:52,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:54:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-07 09:54:52,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:52,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-07 09:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:52,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:52,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:52,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:52,107 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:52,107 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:52,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:52,110 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:52,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:52,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:52,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:52,110 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:52,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:52,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,121 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:52,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,121 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,121 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861292129, max timestamp=1680861292135Num uniques 859; 2023-04-07 09:54:52,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861292129, max timestamp=1680861292135Num uniques 859; 2023-04-07 09:54:52,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:52,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:52,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861292129, max timestamp=1680861292135Num uniques 859; 2023-04-07 09:54:52,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:52,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861352164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861352165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861352166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,221 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:52,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:52,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:52,226 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:52,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:52,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:52,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:52,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:52,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:52,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:52,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@420b8f5d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:52,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:52,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,237 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:52,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:52,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:52,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:52,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:52,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:52,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:52,245 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:52,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:52,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:52,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:52,245 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:52,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:52,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,260 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:52,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:52,261 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:52,261 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:52,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861352269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861352269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861352269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,361 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:52,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:52,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:52,368 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:52,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:52,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:52,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,369 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:52,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:52,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:52,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:52,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:52,371 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:52,371 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,371 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:52,372 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:52,372 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:52,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:52,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:52,372 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:52,373 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,373 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,374 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,374 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:52,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dec3acf[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:52,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:52,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,380 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,384 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,384 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:52,389 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:52,395 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:52,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:52,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:52,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:52,396 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:52,401 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,401 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,402 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,404 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,407 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,410 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,410 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,410 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:52,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:52,410 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,411 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,411 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,411 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,413 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:52,413 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:52,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861352473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861352474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861352474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,514 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:52,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:52,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:52,517 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:52,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:52,517 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:52,518 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,518 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,519 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,519 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,519 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,519 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:52,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:52,520 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:52,520 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,520 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:52,520 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:52,520 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:52,520 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,520 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:52,521 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:52,521 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:52,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@508f5290[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:52,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:52,525 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,527 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:52,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:52,528 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,529 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:52,529 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,529 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:52,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:52,531 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:52,531 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:52,531 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:52,531 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:52,531 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:52,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,534 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:52,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:52,539 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,540 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:52,540 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:52,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,539 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:52,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/4aac5fdf15d24755b3c282753f9c460b 2023-04-07 09:54:52,640 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:52,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:52,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:52,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:52,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:52,643 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:52,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:52,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:52,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:52,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,649 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d88a0af[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:52,650 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:52,651 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,653 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:52,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:52,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:52,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:52,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:52,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:52,657 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:52,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:52,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:52,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:52,657 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:52,658 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,658 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,658 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,659 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,660 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:52,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:52,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,663 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:52,663 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:52,664 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:52,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,764 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:52,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:52,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:52,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:52,767 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:52,767 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,767 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:52,767 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:52,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:52,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:52,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:52,773 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,773 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:52,773 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:52,773 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:52,774 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,774 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4367ad24[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:52,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:52,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:52,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:52,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:52,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:52,780 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861352779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861352779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861352779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:52,783 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:52,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:52,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:52,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:52,784 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:52,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:52,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:52,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,794 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,794 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:52,794 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:52,894 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:52,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:52,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:52,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:52,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:52,899 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:52,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:52,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:52,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,900 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:52,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:52,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:52,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:52,903 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:52,903 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:52,903 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:52,912 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,912 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:52,912 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:52,912 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:52,914 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:52,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff1ddc5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:52,917 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:52,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:52,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:52,921 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:52,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:52,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,922 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:52,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:52,925 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:52,925 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:52,925 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:52,925 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:52,925 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:52,926 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,926 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,926 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:52,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:52,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:52,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:52,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:52,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:52,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:52,932 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:52,932 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:52,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/4f0763b1f874420a8b4713ffd52a81b1 2023-04-07 09:54:53,032 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:53,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:53,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:53,035 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:53,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:53,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:53,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,041 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:53,041 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,041 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:53,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:53,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:53,043 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,043 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:53,043 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:53,043 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:53,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:53,044 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,045 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,045 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:53,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45400979[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:53,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:53,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:53,047 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,047 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:53,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:53,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:53,050 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:53,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:53,051 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:53,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:53,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:53,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:53,052 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:53,053 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,053 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,053 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:53,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:53,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,057 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:53,057 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,057 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:53,157 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:53,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:53,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:53,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:53,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:53,160 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:53,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:53,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:53,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:53,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:53,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:53,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:53,164 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,164 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:53,164 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:53,164 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,167 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:53,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5554a6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:53,169 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:53,169 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:53,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:53,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:53,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:53,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:53,175 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:53,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:53,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:53,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:53,175 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:53,176 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,176 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:53,176 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,181 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:53,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:53,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:53,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,182 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:53,182 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:53,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:53,282 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:53,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:53,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:53,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:53,285 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:53,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:53,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:53,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:53,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:53,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:53,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861353285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:53,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:53,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:53,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:53,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:53,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:53,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861353287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:53,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2325cca5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:53,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:53,292 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:53,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861353287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,292 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:53,293 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,293 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:53,295 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:53,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:53,297 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:53,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:53,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:53,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:53,297 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:53,299 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,299 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:53,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,299 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:53,299 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:53,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,300 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,302 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,302 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:53,302 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:53,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:53,302 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:53,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:53,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:53,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:53,302 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:53,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,303 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:53,303 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:53,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:53,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/f832c62c6dd5405dabda80db113d28c9 2023-04-07 09:54:53,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/4aac5fdf15d24755b3c282753f9c460b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/4aac5fdf15d24755b3c282753f9c460b 2023-04-07 09:54:53,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/4aac5fdf15d24755b3c282753f9c460b, entries=200, sequenceid=356, filesize=14.3 K 2023-04-07 09:54:53,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/4f0763b1f874420a8b4713ffd52a81b1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/4f0763b1f874420a8b4713ffd52a81b1 2023-04-07 09:54:53,389 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/4f0763b1f874420a8b4713ffd52a81b1, entries=150, sequenceid=356, filesize=11.9 K 2023-04-07 09:54:53,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/f832c62c6dd5405dabda80db113d28c9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/f832c62c6dd5405dabda80db113d28c9 2023-04-07 09:54:53,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/f832c62c6dd5405dabda80db113d28c9, entries=150, sequenceid=356, filesize=11.9 K 2023-04-07 09:54:53,396 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~164.95 KB/168912, currentSize=134.18 KB/137400 for 243bf66836bfdaa40ee21a1c79eaec90 in 1291ms, sequenceid=356, compaction requested=true 2023-04-07 09:54:53,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:53,396 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:53,396 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:53,396 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:53,396 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:53,396 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:53,397 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:53,397 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37236 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:53,397 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:53,398 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:53,398 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,398 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,398 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2b23888fccdf4ca9b73708b970ae2557, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2a069cb6fb5845a5b7b223bccdcf7242, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/4f0763b1f874420a8b4713ffd52a81b1] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=34.0 K 2023-04-07 09:54:53,398 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ec5ac61e20dc4be58fa32a769e97fb84, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/71d55da6139f49c697c7b70789358a60, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/4aac5fdf15d24755b3c282753f9c460b] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=36.4 K 2023-04-07 09:54:53,398 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2b23888fccdf4ca9b73708b970ae2557, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680861289166 2023-04-07 09:54:53,398 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ec5ac61e20dc4be58fa32a769e97fb84, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680861289166 2023-04-07 09:54:53,399 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2a069cb6fb5845a5b7b223bccdcf7242, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680861290294 2023-04-07 09:54:53,399 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 71d55da6139f49c697c7b70789358a60, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680861290294 2023-04-07 09:54:53,399 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4f0763b1f874420a8b4713ffd52a81b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861292101 2023-04-07 09:54:53,399 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4aac5fdf15d24755b3c282753f9c460b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861292101 2023-04-07 09:54:53,403 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:53,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:53,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:53,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:53,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:53,406 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:53,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:53,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:53,407 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,407 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,408 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:53,408 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:53,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:53,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:53,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:53,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:53,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:53,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:53,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:53,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:53,424 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-07 09:54:53,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:53,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:53,425 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:53,425 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:53,425 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:53,425 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:53,427 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#300 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:53,429 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#301 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:53,488 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/a4f8019f653b4780bcdbc84d640f2529 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4f8019f653b4780bcdbc84d640f2529 2023-04-07 09:54:53,491 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/793bcae508bb4be5b6b97c7847d89c7a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/793bcae508bb4be5b6b97c7847d89c7a 2023-04-07 09:54:53,497 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 793bcae508bb4be5b6b97c7847d89c7a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:53,497 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:53,497 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861293396; duration=0sec 2023-04-07 09:54:53,497 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:53,497 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:53,499 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into a4f8019f653b4780bcdbc84d640f2529(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:53,499 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:53,499 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861293396; duration=0sec 2023-04-07 09:54:53,499 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:53,499 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:53,499 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:53,499 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:53,499 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/15710071aec04ecab21356b5b848bf6d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/90e49a48f9b34933b1144940a7b1097b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/f832c62c6dd5405dabda80db113d28c9] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=34.0 K 2023-04-07 09:54:53,500 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 15710071aec04ecab21356b5b848bf6d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680861289166 2023-04-07 09:54:53,500 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/268394d0e12a4ed8bfd793a6692ada77 2023-04-07 09:54:53,500 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 90e49a48f9b34933b1144940a7b1097b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680861290294 2023-04-07 09:54:53,501 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting f832c62c6dd5405dabda80db113d28c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861292101 2023-04-07 09:54:53,517 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#304 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:53,537 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/1ab37ffff02646d4abad5b2d5c736db7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/1ab37ffff02646d4abad5b2d5c736db7 2023-04-07 09:54:53,541 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 1ab37ffff02646d4abad5b2d5c736db7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:53,541 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:53,541 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=13, startTime=1680861293396; duration=0sec 2023-04-07 09:54:53,541 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:53,930 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/9070762c3f5544b2ba2a1c606ca58440 2023-04-07 09:54:54,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:54,127 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:54,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861354144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861354147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861354248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861354251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861354291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861354296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861354298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,339 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/2b08ab792e974a5a970c2545e83c4bfc 2023-04-07 09:54:54,343 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/268394d0e12a4ed8bfd793a6692ada77 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/268394d0e12a4ed8bfd793a6692ada77 2023-04-07 09:54:54,347 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/268394d0e12a4ed8bfd793a6692ada77, entries=150, sequenceid=392, filesize=11.9 K 2023-04-07 09:54:54,348 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/9070762c3f5544b2ba2a1c606ca58440 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9070762c3f5544b2ba2a1c606ca58440 2023-04-07 09:54:54,350 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9070762c3f5544b2ba2a1c606ca58440, entries=150, sequenceid=392, filesize=11.9 K 2023-04-07 09:54:54,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/2b08ab792e974a5a970c2545e83c4bfc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2b08ab792e974a5a970c2545e83c4bfc 2023-04-07 09:54:54,354 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2b08ab792e974a5a970c2545e83c4bfc, entries=150, sequenceid=392, filesize=11.9 K 2023-04-07 09:54:54,355 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 243bf66836bfdaa40ee21a1c79eaec90 in 931ms, sequenceid=392, compaction requested=false 2023-04-07 09:54:54,355 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:54,355 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:54,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@425b32f1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:54,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:54,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,364 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,364 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:54,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:54,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:54,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:54,368 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:54,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:54,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:54,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:54,369 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:54,370 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,370 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,370 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,381 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,381 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,381 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:54,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:54,381 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,381 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,382 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:54,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,382 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:54,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:54:54,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:54,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:54,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:54,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:54,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:54,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:54,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/1194aed7ae764fa483b2aa3a18bbbdf5 2023-04-07 09:54:54,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/8f8f881831434829b8a696022422aa48 2023-04-07 09:54:54,483 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:54,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:54,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:54,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:54,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:54,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:54,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:54,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:54,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:54,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:54,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:54,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:54,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:54,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861354487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861294481, max timestamp=1680861294488Num uniques 859; 2023-04-07 09:54:54,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:54,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861294481, max timestamp=1680861294488Num uniques 859; 2023-04-07 09:54:54,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861294481, max timestamp=1680861294488Num uniques 859; 2023-04-07 09:54:54,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:54,490 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:54,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:54,490 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:54,490 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:54,490 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:54,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715e002[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:54,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:54,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,495 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:54,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:54,495 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,496 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:54,500 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:54,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:54,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:54,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:54,500 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:54,502 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,502 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,502 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,503 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,504 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:54,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:54,506 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,507 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:54,506 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,507 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:54,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,507 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48308 deadline: 1680861354535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861354590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,607 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:54,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:54,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:54,610 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:54,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:54,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:54,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:54,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:54,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:54,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70926422[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:54,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,619 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,619 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:54,619 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:54,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:54,623 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:54,623 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:54,623 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:54,623 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:54,623 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:54,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:54,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,630 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:54,630 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:54,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861354638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,730 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:54,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:54,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:54,733 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:54,733 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,733 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:54,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:54,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,735 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,735 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,735 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:54,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:54,737 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,737 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:54,737 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:54,737 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:54,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:54,738 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:54,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5fde71[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:54,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:54,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:54,743 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,743 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:54,749 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:54,751 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:54,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:54,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:54,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:54,751 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:54,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,753 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,755 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:54,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:54,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,758 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:54,757 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,758 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,758 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,758 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,758 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,758 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:54,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:54,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48354 deadline: 1680861354795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861354839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,858 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:54,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:54,861 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:54,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:54,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:54,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,868 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:54,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:54,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:54,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:54,869 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,869 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:54,870 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:54,870 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:54,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6688a791[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:54,873 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:54,873 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:54,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:54,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,876 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:54,878 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:54,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:54,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:54,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:54,878 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:54,880 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,880 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:54,880 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:54,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,881 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/426fa08d8f884e458bff6e73a7f76fa8 2023-04-07 09:54:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:54,882 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,886 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:54,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:54,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:54,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:54,887 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:54,887 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:54,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:54,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/1194aed7ae764fa483b2aa3a18bbbdf5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1194aed7ae764fa483b2aa3a18bbbdf5 2023-04-07 09:54:54,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:54,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1194aed7ae764fa483b2aa3a18bbbdf5, entries=200, sequenceid=406, filesize=14.3 K 2023-04-07 09:54:54,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/8f8f881831434829b8a696022422aa48 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/8f8f881831434829b8a696022422aa48 2023-04-07 09:54:54,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/8f8f881831434829b8a696022422aa48, entries=150, sequenceid=406, filesize=11.9 K 2023-04-07 09:54:54,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/426fa08d8f884e458bff6e73a7f76fa8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/426fa08d8f884e458bff6e73a7f76fa8 2023-04-07 09:54:54,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/426fa08d8f884e458bff6e73a7f76fa8, entries=150, sequenceid=406, filesize=11.9 K 2023-04-07 09:54:54,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 243bf66836bfdaa40ee21a1c79eaec90 in 448ms, sequenceid=406, compaction requested=true 2023-04-07 09:54:54,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:54,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:54,901 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:54,901 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:54,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:54,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:54,902 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39782 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:54,902 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:54,902 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:54,902 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:54,902 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,902 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,902 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/793bcae508bb4be5b6b97c7847d89c7a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/268394d0e12a4ed8bfd793a6692ada77, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1194aed7ae764fa483b2aa3a18bbbdf5] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=38.8 K 2023-04-07 09:54:54,902 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4f8019f653b4780bcdbc84d640f2529, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9070762c3f5544b2ba2a1c606ca58440, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/8f8f881831434829b8a696022422aa48] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=36.5 K 2023-04-07 09:54:54,903 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 793bcae508bb4be5b6b97c7847d89c7a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861292101 2023-04-07 09:54:54,903 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a4f8019f653b4780bcdbc84d640f2529, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861292101 2023-04-07 09:54:54,903 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9070762c3f5544b2ba2a1c606ca58440, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680861292150 2023-04-07 09:54:54,903 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 268394d0e12a4ed8bfd793a6692ada77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680861292150 2023-04-07 09:54:54,904 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8f8f881831434829b8a696022422aa48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680861294141 2023-04-07 09:54:54,904 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1194aed7ae764fa483b2aa3a18bbbdf5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680861294139 2023-04-07 09:54:54,912 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:54,913 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#310 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:54,987 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:54,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:54,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:54,990 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:54,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:54,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:54,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,992 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:54,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:54,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:54,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:54,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:54,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:54,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:54,994 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:54,994 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:54,994 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-07 09:54:54,994 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:54,994 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:54,995 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:54,995 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:54,995 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:54,995 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:55,002 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/494a440356ab49e58faf036403b1c94e 2023-04-07 09:54:55,010 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/212c9f654ca74f7c89c8a364d602e58c 2023-04-07 09:54:55,026 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/68f8af648a174056bdac9c2a64d937dd 2023-04-07 09:54:55,031 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/494a440356ab49e58faf036403b1c94e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/494a440356ab49e58faf036403b1c94e 2023-04-07 09:54:55,034 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/494a440356ab49e58faf036403b1c94e, entries=150, sequenceid=442, filesize=11.9 K 2023-04-07 09:54:55,035 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/212c9f654ca74f7c89c8a364d602e58c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/212c9f654ca74f7c89c8a364d602e58c 2023-04-07 09:54:55,046 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/212c9f654ca74f7c89c8a364d602e58c, entries=150, sequenceid=442, filesize=11.9 K 2023-04-07 09:54:55,047 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/68f8af648a174056bdac9c2a64d937dd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/68f8af648a174056bdac9c2a64d937dd 2023-04-07 09:54:55,051 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/68f8af648a174056bdac9c2a64d937dd, entries=150, sequenceid=442, filesize=11.9 K 2023-04-07 09:54:55,052 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 243bf66836bfdaa40ee21a1c79eaec90 in 58ms, sequenceid=442, compaction requested=true 2023-04-07 09:54:55,052 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:55,052 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,052 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:55,053 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:55,053 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:55,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb54eb2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:55,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:55,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:55,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:55,062 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:55,062 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,062 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:55,066 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:55,066 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:55,066 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:55,066 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:55,066 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:55,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,068 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,071 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,072 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:55,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:55,072 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,072 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,072 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:55,072 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:55,072 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,072 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,072 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:55,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:55,112 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:54:55,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:55,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:55,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:55,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:55,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:55,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:55,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/a5bd4b48835e4a608bc0cff3983211d9 2023-04-07 09:54:55,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/a69d1d5d6a31413e8e03d7c18b52fc07 2023-04-07 09:54:55,172 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:55,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861295148, max timestamp=1680861295170Num uniques 859; 2023-04-07 09:54:55,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861295148, max timestamp=1680861295170Num uniques 859; 2023-04-07 09:54:55,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861295148, max timestamp=1680861295170Num uniques 859; 2023-04-07 09:54:55,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:55,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:55,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:55,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:54:55,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:55,176 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:55,176 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,176 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:55,176 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:55,176 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,176 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:55,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:55,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:55,181 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,181 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:55,181 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:55,181 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:55,182 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:55,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:55,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:55,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d658032[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:55,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:55,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:55,187 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,187 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:55,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:55,191 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:55,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:55,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:55,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:55,191 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:55,195 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,195 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,195 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,196 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,197 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:55,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:55,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:55,200 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,200 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:55,200 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861355205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861355205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,300 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:55,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:55,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:55,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:55,304 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:55,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:55,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:55,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:55,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:55,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:55,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:55,308 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,308 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:55,308 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:55,308 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:55,312 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e4a7fb9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:55,312 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861355309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861355309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,314 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:55,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:55,315 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:55,317 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:55,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:55,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:55,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:55,317 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:55,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,319 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,321 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:55,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,323 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,323 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:55,324 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:55,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,326 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/ef57bec06ecf4f6ba995468f81b6a69d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/ef57bec06ecf4f6ba995468f81b6a69d 2023-04-07 09:54:55,327 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/a9f3c645576143e5804225257ab3a175 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a9f3c645576143e5804225257ab3a175 2023-04-07 09:54:55,336 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into ef57bec06ecf4f6ba995468f81b6a69d(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:55,336 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:55,336 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861294901; duration=0sec 2023-04-07 09:54:55,336 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:55,336 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into a9f3c645576143e5804225257ab3a175(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:55,336 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:55,336 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:55,336 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861294901; duration=0sec 2023-04-07 09:54:55,336 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:55,337 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:55,337 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:55,337 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,337 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/1ab37ffff02646d4abad5b2d5c736db7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2b08ab792e974a5a970c2545e83c4bfc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/426fa08d8f884e458bff6e73a7f76fa8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/68f8af648a174056bdac9c2a64d937dd] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=48.4 K 2023-04-07 09:54:55,338 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1ab37ffff02646d4abad5b2d5c736db7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861292101 2023-04-07 09:54:55,338 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2b08ab792e974a5a970c2545e83c4bfc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680861292150 2023-04-07 09:54:55,338 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 426fa08d8f884e458bff6e73a7f76fa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680861294141 2023-04-07 09:54:55,339 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 68f8af648a174056bdac9c2a64d937dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680861294524 2023-04-07 09:54:55,347 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#317 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:55,424 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:55,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:55,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:55,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:55,427 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:55,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:55,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:55,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,429 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,430 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,430 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:55,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:55,431 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,431 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:55,431 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:55,432 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,433 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:55,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dddf1ce[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:55,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,436 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:55,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:55,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:55,441 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:55,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:55,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:55,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:55,441 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:55,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,444 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,447 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:55,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:55,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,448 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:55,448 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:55,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861355514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861355514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,548 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:55,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:55,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:55,551 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:55,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:55,554 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,554 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:55,554 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:55,555 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:55,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f9fe19[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:55,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:55,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:55,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:55,563 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:55,565 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:55,565 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:55,565 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:55,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/e37236f6056b479ca5c7ca461c343fb4 2023-04-07 09:54:55,565 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:55,565 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:55,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:55,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/a5bd4b48835e4a608bc0cff3983211d9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a5bd4b48835e4a608bc0cff3983211d9 2023-04-07 09:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a5bd4b48835e4a608bc0cff3983211d9, entries=200, sequenceid=452, filesize=14.3 K 2023-04-07 09:54:55,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/a69d1d5d6a31413e8e03d7c18b52fc07 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a69d1d5d6a31413e8e03d7c18b52fc07 2023-04-07 09:54:55,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:55,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:55,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:55,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:55,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,577 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:55,577 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:55,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:55,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a69d1d5d6a31413e8e03d7c18b52fc07, entries=150, sequenceid=452, filesize=11.9 K 2023-04-07 09:54:55,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/e37236f6056b479ca5c7ca461c343fb4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e37236f6056b479ca5c7ca461c343fb4 2023-04-07 09:54:55,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e37236f6056b479ca5c7ca461c343fb4, entries=150, sequenceid=452, filesize=11.9 K 2023-04-07 09:54:55,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 243bf66836bfdaa40ee21a1c79eaec90 in 473ms, sequenceid=452, compaction requested=true 2023-04-07 09:54:55,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:55,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:55,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:55,585 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:55,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:54:55,586 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:55,586 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:55,586 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,586 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a9f3c645576143e5804225257ab3a175, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/494a440356ab49e58faf036403b1c94e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a5bd4b48835e4a608bc0cff3983211d9] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=38.9 K 2023-04-07 09:54:55,587 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting a9f3c645576143e5804225257ab3a175, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680861294141 2023-04-07 09:54:55,587 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 494a440356ab49e58faf036403b1c94e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680861294524 2023-04-07 09:54:55,587 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting a5bd4b48835e4a608bc0cff3983211d9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680861295104 2023-04-07 09:54:55,592 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:55,677 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:55,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:55,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:55,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:55,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:55,680 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:55,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:55,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:55,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:55,682 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:55,682 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,682 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:55,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:55,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:55,684 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:55,684 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:55,691 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/b08bf74444904509bc04c2957f31fda6 2023-04-07 09:54:55,755 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/6c175fc5a1784e9aa1f80a276b762ed1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6c175fc5a1784e9aa1f80a276b762ed1 2023-04-07 09:54:55,759 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 6c175fc5a1784e9aa1f80a276b762ed1(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:55,759 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:55,759 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=12, startTime=1680861294901; duration=0sec 2023-04-07 09:54:55,759 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:54:55,759 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:55,760 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:55,760 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:55,760 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:55,760 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/ef57bec06ecf4f6ba995468f81b6a69d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/212c9f654ca74f7c89c8a364d602e58c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a69d1d5d6a31413e8e03d7c18b52fc07] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=36.6 K 2023-04-07 09:54:55,761 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ef57bec06ecf4f6ba995468f81b6a69d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680861294141 2023-04-07 09:54:55,761 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 212c9f654ca74f7c89c8a364d602e58c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680861294524 2023-04-07 09:54:55,761 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a69d1d5d6a31413e8e03d7c18b52fc07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680861295107 2023-04-07 09:54:55,777 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:55,790 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/03aadf89477743b38fafe0ae9726b1b4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/03aadf89477743b38fafe0ae9726b1b4 2023-04-07 09:54:55,794 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into 03aadf89477743b38fafe0ae9726b1b4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:55,794 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:55,794 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861295585; duration=0sec 2023-04-07 09:54:55,794 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:55,794 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-07 09:54:55,795 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-07 09:54:55,795 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-07 09:54:55,795 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. because compaction request was cancelled 2023-04-07 09:54:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:55,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:55,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861355825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861355826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861355928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:55,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:55,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861355929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,001 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/1c9b5299543a4502a552c4959aa43884 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1c9b5299543a4502a552c4959aa43884 2023-04-07 09:54:56,005 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 1c9b5299543a4502a552c4959aa43884(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:56,005 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:56,005 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861295585; duration=0sec 2023-04-07 09:54:56,005 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:56,103 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/108125c6e43f48c4a9300483b1918b37 2023-04-07 09:54:56,125 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/a559748d7be546fea6c2005d2d44c1b0 2023-04-07 09:54:56,130 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/b08bf74444904509bc04c2957f31fda6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b08bf74444904509bc04c2957f31fda6 2023-04-07 09:54:56,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861356132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861356132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,142 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b08bf74444904509bc04c2957f31fda6, entries=150, sequenceid=490, filesize=11.9 K 2023-04-07 09:54:56,143 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/108125c6e43f48c4a9300483b1918b37 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/108125c6e43f48c4a9300483b1918b37 2023-04-07 09:54:56,148 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/108125c6e43f48c4a9300483b1918b37, entries=150, sequenceid=490, filesize=11.9 K 2023-04-07 09:54:56,148 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/a559748d7be546fea6c2005d2d44c1b0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/a559748d7be546fea6c2005d2d44c1b0 2023-04-07 09:54:56,154 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/a559748d7be546fea6c2005d2d44c1b0, entries=150, sequenceid=490, filesize=11.9 K 2023-04-07 09:54:56,155 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 243bf66836bfdaa40ee21a1c79eaec90 in 471ms, sequenceid=490, compaction requested=true 2023-04-07 09:54:56,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:56,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:56,156 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:56,156 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,156 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:56,156 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:56,157 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:56,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@689c8704[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:56,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:56,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,162 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:56,162 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:56,162 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:56,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:56,167 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:56,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:56,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:56,169 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:56,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:56,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:56,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:56,170 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:56,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:56,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:56,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,178 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:56,178 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:56,278 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:56,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:56,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:56,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:56,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:56,282 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:56,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:56,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:56,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,283 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,283 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,284 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,284 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:56,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:56,285 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:56,285 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,285 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:56,285 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:56,285 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:56,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:56,285 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:54:56,285 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:56,286 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:56,286 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:56,286 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:56,286 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:56,286 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:56,294 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/6b56cd8a2fa84fcb8ad0fd440468671e 2023-04-07 09:54:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:56,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:56,318 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/520ad3ef3f0e40218b2a7dddf23f4a71 2023-04-07 09:54:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861296319, max timestamp=1680861296336Num uniques 859; 2023-04-07 09:54:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861296319, max timestamp=1680861296336Num uniques 859; 2023-04-07 09:54:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:54:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861296319, max timestamp=1680861296336Num uniques 859; 2023-04-07 09:54:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:54:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:54:56,360 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/cdc73fae0c5840c7a1349c8eb3691bc5 2023-04-07 09:54:56,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861356359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861356360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861356362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,371 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/6b56cd8a2fa84fcb8ad0fd440468671e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/6b56cd8a2fa84fcb8ad0fd440468671e 2023-04-07 09:54:56,377 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/6b56cd8a2fa84fcb8ad0fd440468671e, entries=150, sequenceid=503, filesize=11.9 K 2023-04-07 09:54:56,378 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/520ad3ef3f0e40218b2a7dddf23f4a71 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/520ad3ef3f0e40218b2a7dddf23f4a71 2023-04-07 09:54:56,381 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/520ad3ef3f0e40218b2a7dddf23f4a71, entries=150, sequenceid=503, filesize=11.9 K 2023-04-07 09:54:56,382 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/cdc73fae0c5840c7a1349c8eb3691bc5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/cdc73fae0c5840c7a1349c8eb3691bc5 2023-04-07 09:54:56,386 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/cdc73fae0c5840c7a1349c8eb3691bc5, entries=150, sequenceid=503, filesize=11.9 K 2023-04-07 09:54:56,386 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 243bf66836bfdaa40ee21a1c79eaec90 in 101ms, sequenceid=503, compaction requested=true 2023-04-07 09:54:56,387 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:56,387 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:56,390 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:56,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d68a19[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:56,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:56,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,393 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:56,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:56,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:56,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:56,395 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:56,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:56,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:56,399 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:56,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:56,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:56,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:56,399 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:56,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:56,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,406 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:56,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,407 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:56,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:56,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-07 09:54:56,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:56,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:56,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:56,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:56,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:56,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:54:56,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/785d7240e34b4235b12a9cde0d3b834d 2023-04-07 09:54:56,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861356451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861356453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861356464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861356466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861356466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/d4f5720f6c1a44739057e406b48d639c 2023-04-07 09:54:56,507 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:56,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:56,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:56,511 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:56,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:56,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:56,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,513 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:56,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:56,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:56,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:56,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:56,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:56,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:56,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:56,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77e74aec[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:56,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:56,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:56,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:56,524 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:56,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:56,530 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:56,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:56,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:56,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:56,530 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:56,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:56,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:56,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:56,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,542 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,542 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,541 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,542 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:56,542 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:56,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861356559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861356565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,642 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:56,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:56,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:56,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:56,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:56,646 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:56,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:56,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:56,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,647 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:56,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:56,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:56,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:56,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:56,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:56,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:56,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:56,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:56,652 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51486b3f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:56,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:56,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:56,657 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:56,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:56,662 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:56,662 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:56,662 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:56,662 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:56,662 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:56,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861356667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861356671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861356672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,681 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,682 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:56,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:56,682 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:56,682 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:56,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861356765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:56,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861356770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,782 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:56,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:56,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:56,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:56,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:56,786 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:56,786 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,786 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:56,786 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:56,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,788 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:56,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:56,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:56,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:56,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:56,794 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,794 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:56,794 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:56,794 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:56,795 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6062b5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:56,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:56,798 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:56,802 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:56,802 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,802 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:56,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:56,804 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:56,805 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:56,805 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:56,805 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:56,805 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:56,806 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,806 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,806 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:56,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:56,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:56,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,811 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:56,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:56,811 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:56,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:56,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:56,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,887 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/784b19451ea74ed98cfb7487840932c3 2023-04-07 09:54:56,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/785d7240e34b4235b12a9cde0d3b834d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/785d7240e34b4235b12a9cde0d3b834d 2023-04-07 09:54:56,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/785d7240e34b4235b12a9cde0d3b834d, entries=150, sequenceid=539, filesize=11.9 K 2023-04-07 09:54:56,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/d4f5720f6c1a44739057e406b48d639c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d4f5720f6c1a44739057e406b48d639c 2023-04-07 09:54:56,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d4f5720f6c1a44739057e406b48d639c, entries=150, sequenceid=539, filesize=11.9 K 2023-04-07 09:54:56,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/784b19451ea74ed98cfb7487840932c3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/784b19451ea74ed98cfb7487840932c3 2023-04-07 09:54:56,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/784b19451ea74ed98cfb7487840932c3, entries=150, sequenceid=539, filesize=11.9 K 2023-04-07 09:54:56,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 243bf66836bfdaa40ee21a1c79eaec90 in 468ms, sequenceid=539, compaction requested=true 2023-04-07 09:54:56,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:56,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:56,906 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:56,906 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:54:56,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:56,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:56,907 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:56,907 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:56,907 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:54:56,907 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,907 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:56,907 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1c9b5299543a4502a552c4959aa43884, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b08bf74444904509bc04c2957f31fda6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/6b56cd8a2fa84fcb8ad0fd440468671e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/785d7240e34b4235b12a9cde0d3b834d] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=48.6 K 2023-04-07 09:54:56,907 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,908 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/03aadf89477743b38fafe0ae9726b1b4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/108125c6e43f48c4a9300483b1918b37, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/520ad3ef3f0e40218b2a7dddf23f4a71, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d4f5720f6c1a44739057e406b48d639c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=48.6 K 2023-04-07 09:54:56,908 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1c9b5299543a4502a552c4959aa43884, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680861295107 2023-04-07 09:54:56,908 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 03aadf89477743b38fafe0ae9726b1b4, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680861295107 2023-04-07 09:54:56,908 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b08bf74444904509bc04c2957f31fda6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680861295200 2023-04-07 09:54:56,908 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 108125c6e43f48c4a9300483b1918b37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680861295200 2023-04-07 09:54:56,909 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 6b56cd8a2fa84fcb8ad0fd440468671e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680861295817 2023-04-07 09:54:56,909 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 520ad3ef3f0e40218b2a7dddf23f4a71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680861295817 2023-04-07 09:54:56,909 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 785d7240e34b4235b12a9cde0d3b834d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861296357 2023-04-07 09:54:56,909 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d4f5720f6c1a44739057e406b48d639c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861296357 2023-04-07 09:54:56,912 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:56,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:56,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:56,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:56,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:56,915 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:56,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:56,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:56,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:56,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,919 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:56,919 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:56,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:56,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:56,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:56,921 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#329 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:56,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:56,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:56,922 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,922 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:56,922 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-07 09:54:56,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:56,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:56,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:56,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:56,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:56,923 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:56,950 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#330 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:56,957 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/cb27048c940f435f997cb77cea704875 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/cb27048c940f435f997cb77cea704875 2023-04-07 09:54:56,964 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into cb27048c940f435f997cb77cea704875(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:56,964 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:56,964 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=12, startTime=1680861296906; duration=0sec 2023-04-07 09:54:56,964 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:56,964 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:54:56,968 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:54:56,968 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:56,968 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:56,968 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6c175fc5a1784e9aa1f80a276b762ed1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e37236f6056b479ca5c7ca461c343fb4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/a559748d7be546fea6c2005d2d44c1b0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/cdc73fae0c5840c7a1349c8eb3691bc5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/784b19451ea74ed98cfb7487840932c3] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=60.4 K 2023-04-07 09:54:56,969 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 6c175fc5a1784e9aa1f80a276b762ed1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680861294524 2023-04-07 09:54:56,969 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e37236f6056b479ca5c7ca461c343fb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680861295107 2023-04-07 09:54:56,970 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a559748d7be546fea6c2005d2d44c1b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680861295200 2023-04-07 09:54:56,970 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cdc73fae0c5840c7a1349c8eb3691bc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680861295817 2023-04-07 09:54:56,970 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 784b19451ea74ed98cfb7487840932c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861296357 2023-04-07 09:54:56,975 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/1f76dd50852840ff8382e67c13af8c3d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1f76dd50852840ff8382e67c13af8c3d 2023-04-07 09:54:56,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:56,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:56,982 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 1f76dd50852840ff8382e67c13af8c3d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:56,982 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:56,982 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=12, startTime=1680861296906; duration=0sec 2023-04-07 09:54:56,982 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:56,984 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#332 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:57,000 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/6ec6375f32794f30a7e80f7783042b38 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6ec6375f32794f30a7e80f7783042b38 2023-04-07 09:54:57,030 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 6ec6375f32794f30a7e80f7783042b38(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:57,030 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:57,030 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=11, startTime=1680861296906; duration=0sec 2023-04-07 09:54:57,030 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:57,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861296995, max timestamp=1680861297036Num uniques 859; 2023-04-07 09:54:57,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:57,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861296995, max timestamp=1680861297036Num uniques 859; 2023-04-07 09:54:57,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:57,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861296995, max timestamp=1680861297037Num uniques 859; 2023-04-07 09:54:57,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:57,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861297056, max timestamp=1680861297062Num uniques 1018; 2023-04-07 09:54:57,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861297056, max timestamp=1680861297062Num uniques 1018; 2023-04-07 09:54:57,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861297056, max timestamp=1680861297062Num uniques 1018; 2023-04-07 09:54:57,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:57,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861357061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:57,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:57,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680861297102, max timestamp=1680861297109Num uniques 1027; 2023-04-07 09:54:57,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-07 09:54:57,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680861297102, max timestamp=1680861297109Num uniques 1027; 2023-04-07 09:54:57,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-07 09:54:57,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680861297102, max timestamp=1680861297109Num uniques 1027; 2023-04-07 09:54:57,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-07 09:54:57,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861357126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861357127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861357128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861357128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861357166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861357230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861357231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861357231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861357231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861357368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,375 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/c23968a358d4400a881d939dad673a6b 2023-04-07 09:54:57,384 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/db1df12d5fd645949685c32dcdbc6979 2023-04-07 09:54:57,394 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/dbf120fb687d4ed2a74fc3d4f27e8bd3 2023-04-07 09:54:57,398 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/c23968a358d4400a881d939dad673a6b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/c23968a358d4400a881d939dad673a6b 2023-04-07 09:54:57,401 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/c23968a358d4400a881d939dad673a6b, entries=150, sequenceid=548, filesize=11.9 K 2023-04-07 09:54:57,402 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/db1df12d5fd645949685c32dcdbc6979 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/db1df12d5fd645949685c32dcdbc6979 2023-04-07 09:54:57,405 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/db1df12d5fd645949685c32dcdbc6979, entries=150, sequenceid=548, filesize=11.9 K 2023-04-07 09:54:57,406 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/dbf120fb687d4ed2a74fc3d4f27e8bd3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dbf120fb687d4ed2a74fc3d4f27e8bd3 2023-04-07 09:54:57,409 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dbf120fb687d4ed2a74fc3d4f27e8bd3, entries=150, sequenceid=548, filesize=11.9 K 2023-04-07 09:54:57,410 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 243bf66836bfdaa40ee21a1c79eaec90 in 488ms, sequenceid=548, compaction requested=false 2023-04-07 09:54:57,410 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:57,410 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,410 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:57,410 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:57,410 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,410 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:57,410 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:57,412 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:57,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cc7bd42[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:57,413 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:57,413 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,414 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,414 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:57,414 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:57,414 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:57,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:57,416 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:57,418 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:57,418 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:57,418 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:57,418 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:57,418 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:57,420 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,421 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,421 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,421 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,424 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,425 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:57,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:57,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:57,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,425 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:57,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,425 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:57,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:57,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=154.31 KB heapSize=434.39 KB 2023-04-07 09:54:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:57,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:57,451 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/0d2fc0f101204c5b80a82ed52f9c878c 2023-04-07 09:54:57,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861357446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861357465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861357465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861357465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,525 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:57,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:57,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:57,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:57,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:57,528 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:57,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:57,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:57,529 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,529 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,530 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:57,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:57,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:57,531 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,531 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:57,531 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:57,531 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,531 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:57,532 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:57,532 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,532 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:57,532 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:57,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31ba381c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:57,538 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:57,538 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,538 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:57,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:57,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:57,539 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,539 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:57,540 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:57,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:57,542 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:57,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:57,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:57,542 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:57,542 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:57,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,544 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:57,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:57,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,549 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:57,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:57,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,549 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:57,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861357567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861357567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861357568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861357568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,649 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:57,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:57,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:57,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:57,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:57,652 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:57,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:57,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:57,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,655 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,655 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:57,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:57,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:57,656 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,656 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:57,656 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:57,657 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:57,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:57,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:57,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:57,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,663 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,663 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:57,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b9bc87[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:57,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:57,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,666 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:57,667 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,667 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:57,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861357671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:57,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:57,675 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:57,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:57,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:57,675 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:57,675 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:57,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,679 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:57,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:57,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,683 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,683 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:57,684 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:57,684 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,684 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861357769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861357770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861357771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861357771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,784 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:57,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:57,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:57,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:57,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:57,787 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:57,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:57,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:57,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,789 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:57,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:57,790 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:57,790 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:57,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:57,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:57,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:57,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:57,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:57,794 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:57,795 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d12078f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:57,795 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:57,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:57,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:57,798 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:57,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:57,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:57,800 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:57,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:57,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:57,801 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:57,801 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:57,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:57,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:57,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:57,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,807 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,807 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:57,808 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:57,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/6c2cae7a4d184d16b536e73349d4df86 2023-04-07 09:54:57,908 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:57,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:57,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:57,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:57,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:57,911 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:57,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:57,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:57,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,913 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,913 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:57,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:57,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:57,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:57,915 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,915 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:57,915 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:57,915 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:57,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:57,916 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,917 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,917 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:57,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfe98c2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:57,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:57,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,922 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:57,922 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,922 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:57,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:57,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,923 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:57,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:57,926 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:57,926 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:57,926 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:57,926 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:57,926 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:57,927 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,927 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:57,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,927 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:57,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:57,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:57,930 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:57,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,932 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:57,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:57,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:57,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:57,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:57,932 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:57,932 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:57,932 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,932 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:57,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:57,932 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:57,932 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:57,932 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:58,032 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:58,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:58,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:58,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:58,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:58,036 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:58,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:58,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:58,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:58,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:58,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:58,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:58,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:58,045 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:58,046 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,046 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:58,046 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:58,046 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:58,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:58,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c643d99[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:58,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:58,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,050 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:58,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:58,051 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:58,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:58,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:58,053 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:58,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:58,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:58,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:58,053 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:58,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:58,056 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,058 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:58,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:58,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:58,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:58,060 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,060 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:58,060 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,060 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:58,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:58,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861358073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:58,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861358073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:58,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861358074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:58,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861358074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,160 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:58,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:58,164 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:58,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:58,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:58,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:58,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:58,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:58,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:58,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861358174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:58,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:58,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:58,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:58,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:58,192 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,193 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:58,193 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:58,193 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:58,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:58,194 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,195 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:58,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d7ff16[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:58,197 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:58,197 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,199 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,199 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:58,199 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:58,199 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:58,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:58,201 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:58,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:58,205 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:58,205 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:58,205 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:58,205 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:58,205 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:58,207 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,207 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:58,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,207 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,208 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,210 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,213 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:58,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:58,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:58,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:58,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:58,214 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:58,214 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:58,214 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,214 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:58,214 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:58,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/0f0751e62b6943d4a7b2add4eda4c516 2023-04-07 09:54:58,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/0d2fc0f101204c5b80a82ed52f9c878c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/0d2fc0f101204c5b80a82ed52f9c878c 2023-04-07 09:54:58,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/0d2fc0f101204c5b80a82ed52f9c878c, entries=150, sequenceid=623, filesize=11.9 K 2023-04-07 09:54:58,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/6c2cae7a4d184d16b536e73349d4df86 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/6c2cae7a4d184d16b536e73349d4df86 2023-04-07 09:54:58,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/6c2cae7a4d184d16b536e73349d4df86, entries=150, sequenceid=623, filesize=11.9 K 2023-04-07 09:54:58,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/0f0751e62b6943d4a7b2add4eda4c516 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0f0751e62b6943d4a7b2add4eda4c516 2023-04-07 09:54:58,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0f0751e62b6943d4a7b2add4eda4c516, entries=150, sequenceid=623, filesize=11.9 K 2023-04-07 09:54:58,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~434.34 KB/444768, currentSize=26.84 KB/27480 for 243bf66836bfdaa40ee21a1c79eaec90 in 853ms, sequenceid=623, compaction requested=true 2023-04-07 09:54:58,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:58,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:58,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:58,289 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:58,289 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:58,289 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:54:58,290 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:58,290 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:54:58,290 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:58,291 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,291 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:54:58,291 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1f76dd50852840ff8382e67c13af8c3d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/c23968a358d4400a881d939dad673a6b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/0d2fc0f101204c5b80a82ed52f9c878c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=36.8 K 2023-04-07 09:54:58,291 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,291 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/cb27048c940f435f997cb77cea704875, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/db1df12d5fd645949685c32dcdbc6979, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/6c2cae7a4d184d16b536e73349d4df86] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=36.8 K 2023-04-07 09:54:58,293 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1f76dd50852840ff8382e67c13af8c3d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861296357 2023-04-07 09:54:58,293 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting cb27048c940f435f997cb77cea704875, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861296357 2023-04-07 09:54:58,294 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c23968a358d4400a881d939dad673a6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680861296446 2023-04-07 09:54:58,294 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting db1df12d5fd645949685c32dcdbc6979, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680861296446 2023-04-07 09:54:58,294 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0d2fc0f101204c5b80a82ed52f9c878c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680861297124 2023-04-07 09:54:58,295 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 6c2cae7a4d184d16b536e73349d4df86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680861297124 2023-04-07 09:54:58,303 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#338 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:58,307 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:58,314 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:58,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:58,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:58,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:58,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:58,324 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:58,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:58,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:58,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:58,330 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:58,331 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,331 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:58,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:58,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:58,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:58,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:58,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:58,348 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,348 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:58,350 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-07 09:54:58,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:58,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:58,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:58,351 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:58,354 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:58,354 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:58,355 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/f967e2285fcb4346a3e7473e7300ec46 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f967e2285fcb4346a3e7473e7300ec46 2023-04-07 09:54:58,363 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into f967e2285fcb4346a3e7473e7300ec46(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:58,363 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:58,363 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=13, startTime=1680861298289; duration=0sec 2023-04-07 09:54:58,364 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:54:58,364 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:54:58,365 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:54:58,365 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:54:58,365 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,365 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6ec6375f32794f30a7e80f7783042b38, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dbf120fb687d4ed2a74fc3d4f27e8bd3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0f0751e62b6943d4a7b2add4eda4c516] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=36.8 K 2023-04-07 09:54:58,365 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 6ec6375f32794f30a7e80f7783042b38, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861296357 2023-04-07 09:54:58,366 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting dbf120fb687d4ed2a74fc3d4f27e8bd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1680861296446 2023-04-07 09:54:58,366 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 0f0751e62b6943d4a7b2add4eda4c516, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680861297124 2023-04-07 09:54:58,383 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#341 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:54:58,388 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/e57ea5b483344f97928e12ed3fbe0331 2023-04-07 09:54:58,406 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/9dadba4b93984f09a8786a38e7047d60 2023-04-07 09:54:58,418 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/278fec4af0404d8c9d14a9ee8474c198 2023-04-07 09:54:58,422 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/e57ea5b483344f97928e12ed3fbe0331 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/e57ea5b483344f97928e12ed3fbe0331 2023-04-07 09:54:58,428 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/e57ea5b483344f97928e12ed3fbe0331, entries=150, sequenceid=630, filesize=11.9 K 2023-04-07 09:54:58,429 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/9dadba4b93984f09a8786a38e7047d60 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9dadba4b93984f09a8786a38e7047d60 2023-04-07 09:54:58,433 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9dadba4b93984f09a8786a38e7047d60, entries=150, sequenceid=630, filesize=11.9 K 2023-04-07 09:54:58,434 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/278fec4af0404d8c9d14a9ee8474c198 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/278fec4af0404d8c9d14a9ee8474c198 2023-04-07 09:54:58,440 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/278fec4af0404d8c9d14a9ee8474c198, entries=150, sequenceid=630, filesize=11.9 K 2023-04-07 09:54:58,441 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 243bf66836bfdaa40ee21a1c79eaec90 in 91ms, sequenceid=630, compaction requested=false 2023-04-07 09:54:58,441 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:58,441 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:58,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,442 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:58,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f94a8bc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:58,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:58,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:58,446 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,446 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:58,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:58,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:58,452 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:58,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:58,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:58,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:58,452 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:58,459 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,460 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:58,460 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:58,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:58,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,464 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:58,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:58,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:58,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:58,465 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:58,466 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:58,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:58,566 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:58,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:58,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:58,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:58,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:58,569 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:58,569 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,569 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:58,569 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:58,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:58,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:58,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:58,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:58,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:58,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:58,577 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:58,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:58,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:58,578 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-07 09:54:58,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:58,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:58,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:58,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:58,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:58,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:54:58,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:58,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:58,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861298588, max timestamp=1680861298594Num uniques 859; 2023-04-07 09:54:58,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:58,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861298588, max timestamp=1680861298594Num uniques 859; 2023-04-07 09:54:58,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:58,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861298588, max timestamp=1680861298594Num uniques 859; 2023-04-07 09:54:58,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:54:58,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7683, totalHeapSize=17712, min timestamp=1680861298611, max timestamp=1680861298615Num uniques 1018; 2023-04-07 09:54:58,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861298611, max timestamp=1680861298615Num uniques 1018; 2023-04-07 09:54:58,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46613, new segment data size=7683, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:58,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861298611, max timestamp=1680861298615Num uniques 1018; 2023-04-07 09:54:58,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:58,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1680861298632, max timestamp=1680861298638Num uniques 1027; 2023-04-07 09:54:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=8092, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 09:54:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8095, totalHeapSize=18648, min timestamp=1680861298632, max timestamp=1680861298638Num uniques 1027; 2023-04-07 09:54:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47025, new segment data size=8095, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 09:54:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680861298632, max timestamp=1680861298638Num uniques 1027; 2023-04-07 09:54:58,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 09:54:58,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680861298643, max timestamp=1680861298657Num uniques 1036; 2023-04-07 09:54:58,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-07 09:54:58,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680861298643, max timestamp=1680861298657Num uniques 1036; 2023-04-07 09:54:58,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-07 09:54:58,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680861298643, max timestamp=1680861298657Num uniques 1036; 2023-04-07 09:54:58,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-07 09:54:58,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861298673, max timestamp=1680861298681Num uniques 1145; 2023-04-07 09:54:58,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-07 09:54:58,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861298673, max timestamp=1680861298681Num uniques 1145; 2023-04-07 09:54:58,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-07 09:54:58,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861298676, max timestamp=1680861298681Num uniques 1145; 2023-04-07 09:54:58,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=6870, suffix heap size=147048,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=150 2023-04-07 09:54:58,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680861298717, max timestamp=1680861298722Num uniques 1154; 2023-04-07 09:54:58,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680861298717, max timestamp=1680861298722Num uniques 1009; 2023-04-07 09:54:58,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-07 09:54:58,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680861298717, max timestamp=1680861298722Num uniques 1154; 2023-04-07 09:54:58,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=11630, suffix heap size=132904,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=254 2023-04-07 09:54:58,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-07 09:54:58,731 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/a8f00d344b2040f7ac8605877d582d83 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a8f00d344b2040f7ac8605877d582d83 2023-04-07 09:54:58,735 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into a8f00d344b2040f7ac8605877d582d83(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:58,735 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:58,735 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=13, startTime=1680861298289; duration=0sec 2023-04-07 09:54:58,735 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:58,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680861298733, max timestamp=1680861298745Num uniques 1113; 2023-04-07 09:54:58,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-07 09:54:58,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680861298733, max timestamp=1680861298745Num uniques 1113; 2023-04-07 09:54:58,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680861298733, max timestamp=1680861298745Num uniques 1113; 2023-04-07 09:54:58,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-07 09:54:58,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-07 09:54:58,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680861298757, max timestamp=1680861298763Num uniques 1172; 2023-04-07 09:54:58,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680861298757, max timestamp=1680861298763Num uniques 1172; 2023-04-07 09:54:58,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-07 09:54:58,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-07 09:54:58,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680861298757, max timestamp=1680861298763Num uniques 1172; 2023-04-07 09:54:58,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=12452, suffix heap size=149856,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=272 2023-04-07 09:54:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680861298773, max timestamp=1680861298779Num uniques 1131; 2023-04-07 09:54:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680861298773, max timestamp=1680861298779Num uniques 1131; 2023-04-07 09:54:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-07 09:54:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10566, totalHeapSize=24264, min timestamp=1680861298773, max timestamp=1680861298779Num uniques 1131; 2023-04-07 09:54:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-07 09:54:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51786, new segment data size=10566, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-07 09:54:58,795 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/2ee889f3e12444ff88e93dc1a900a161 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2ee889f3e12444ff88e93dc1a900a161 2023-04-07 09:54:58,799 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 2ee889f3e12444ff88e93dc1a900a161(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:54:58,799 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:58,799 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=13, startTime=1680861298289; duration=0sec 2023-04-07 09:54:58,800 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:54:58,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680861298795, max timestamp=1680861298799Num uniques 1090; 2023-04-07 09:54:58,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-07 09:54:58,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680861298795, max timestamp=1680861298799Num uniques 1090; 2023-04-07 09:54:58,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680861298795, max timestamp=1680861298799Num uniques 1090; 2023-04-07 09:54:58,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-07 09:54:58,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-07 09:54:58,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680861298805, max timestamp=1680861298816Num uniques 1049; 2023-04-07 09:54:58,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 09:54:58,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680861298805, max timestamp=1680861298816Num uniques 1049; 2023-04-07 09:54:58,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680861298805, max timestamp=1680861298816Num uniques 1049; 2023-04-07 09:54:58,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 09:54:58,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-07 09:54:58,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680861298819, max timestamp=1680861298830Num uniques 1058; 2023-04-07 09:54:58,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680861298819, max timestamp=1680861298830Num uniques 1058; 2023-04-07 09:54:58,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-07 09:54:58,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680861298819, max timestamp=1680861298830Num uniques 1058; 2023-04-07 09:54:58,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-07 09:54:58,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-07 09:54:58,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680861298838, max timestamp=1680861298849Num uniques 1117; 2023-04-07 09:54:58,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-07 09:54:58,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680861298838, max timestamp=1680861298849Num uniques 1117; 2023-04-07 09:54:58,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680861298838, max timestamp=1680861298849Num uniques 1117; 2023-04-07 09:54:58,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-07 09:54:58,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-07 09:54:58,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680861298851, max timestamp=1680861298866Num uniques 1076; 2023-04-07 09:54:58,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 09:54:58,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680861298851, max timestamp=1680861298866Num uniques 1076; 2023-04-07 09:54:58,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1680861298851, max timestamp=1680861298866Num uniques 1076; 2023-04-07 09:54:58,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49266, new segment data size=8046, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 09:54:58,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-07 09:54:58,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680861298873, max timestamp=1680861298913Num uniques 1035; 2023-04-07 09:54:58,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680861298873, max timestamp=1680861298913Num uniques 1035; 2023-04-07 09:54:58,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1680861298873, max timestamp=1680861298913Num uniques 1035; 2023-04-07 09:54:58,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-07 09:54:58,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-07 09:54:58,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-07 09:54:58,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1680861298923, max timestamp=1680861298927Num uniques 1144; 2023-04-07 09:54:58,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1680861298923, max timestamp=1680861298927Num uniques 1144; 2023-04-07 09:54:58,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-07 09:54:58,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1680861298923, max timestamp=1680861298927Num uniques 1144; 2023-04-07 09:54:58,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-07 09:54:58,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-07 09:54:58,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680861298937, max timestamp=1680861298947Num uniques 1203; 2023-04-07 09:54:58,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=7005, suffix heap size=153080,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=153 2023-04-07 09:54:58,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680861298937, max timestamp=1680861298947Num uniques 1203; 2023-04-07 09:54:58,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1680861298937, max timestamp=1680861298947Num uniques 1203; 2023-04-07 09:54:58,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=7005, suffix heap size=153080,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=153 2023-04-07 09:54:58,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=7005, suffix heap size=153080,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=153 2023-04-07 09:54:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680861298957, max timestamp=1680861298964Num uniques 1012; 2023-04-07 09:54:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-07 09:54:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7410, totalHeapSize=17088, min timestamp=1680861298957, max timestamp=1680861298965Num uniques 1012; 2023-04-07 09:54:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680861298957, max timestamp=1680861298964Num uniques 1012; 2023-04-07 09:54:58,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46340, new segment data size=7410, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-07 09:54:58,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-04-07 09:54:58,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7822, totalHeapSize=18024, min timestamp=1680861298977, max timestamp=1680861298981Num uniques 1021; 2023-04-07 09:54:58,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46752, new segment data size=7822, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-07 09:54:58,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7816, totalHeapSize=18024, min timestamp=1680861298977, max timestamp=1680861298981Num uniques 1021; 2023-04-07 09:54:58,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46746, new segment data size=7816, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-07 09:54:58,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7818, totalHeapSize=18024, min timestamp=1680861298977, max timestamp=1680861298981Num uniques 1021; 2023-04-07 09:54:58,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46748, new segment data size=7818, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-07 09:54:58,998 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/a1b60be6e9a94c3db88065a912e59ed8 2023-04-07 09:54:59,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680861299017, max timestamp=1680861299022Num uniques 1030; 2023-04-07 09:54:59,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-07 09:54:59,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680861299017, max timestamp=1680861299022Num uniques 1030; 2023-04-07 09:54:59,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-07 09:54:59,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1680861299017, max timestamp=1680861299022Num uniques 1030; 2023-04-07 09:54:59,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-07 09:54:59,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680861299033, max timestamp=1680861299038Num uniques 1139; 2023-04-07 09:54:59,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-07 09:54:59,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680861299033, max timestamp=1680861299038Num uniques 1039; 2023-04-07 09:54:59,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-07 09:54:59,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680861299033, max timestamp=1680861299038Num uniques 1139; 2023-04-07 09:54:59,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-07 09:54:59,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680861299078, max timestamp=1680861299082Num uniques 1098; 2023-04-07 09:54:59,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-07 09:54:59,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680861299078, max timestamp=1680861299082Num uniques 1098; 2023-04-07 09:54:59,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-07 09:54:59,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1680861299078, max timestamp=1680861299082Num uniques 1098; 2023-04-07 09:54:59,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-04-07 09:54:59,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1680861299094, max timestamp=1680861299100Num uniques 1107; 2023-04-07 09:54:59,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-07 09:54:59,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1680861299094, max timestamp=1680861299100Num uniques 1107; 2023-04-07 09:54:59,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=7185, suffix heap size=143096,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=157 2023-04-07 09:54:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861299094, max timestamp=1680861299100Num uniques 1107; 2023-04-07 09:54:59,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=6870, suffix heap size=143096,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=150 2023-04-07 09:54:59,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680861299111, max timestamp=1680861299118Num uniques 1066; 2023-04-07 09:54:59,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-07 09:54:59,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680861299111, max timestamp=1680861299118Num uniques 1016; 2023-04-07 09:54:59,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1680861299111, max timestamp=1680861299118Num uniques 1009; 2023-04-07 09:54:59,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-04-07 09:54:59,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46211, new segment data size=9886, suffix heap size=132904,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=216 2023-04-07 09:54:59,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680861299129, max timestamp=1680861299133Num uniques 1075; 2023-04-07 09:54:59,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-07 09:54:59,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680861299129, max timestamp=1680861299133Num uniques 1075; 2023-04-07 09:54:59,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-07 09:54:59,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1680861299129, max timestamp=1680861299133Num uniques 1075; 2023-04-07 09:54:59,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-07 09:54:59,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680861299146, max timestamp=1680861299151Num uniques 1084; 2023-04-07 09:54:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-07 09:54:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680861299146, max timestamp=1680861299151Num uniques 1084; 2023-04-07 09:54:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1680861299146, max timestamp=1680861299151Num uniques 1084; 2023-04-07 09:54:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-07 09:54:59,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-04-07 09:54:59,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1680861299162, max timestamp=1680861299166Num uniques 1093; 2023-04-07 09:54:59,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50048, new segment data size=11118, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-07 09:54:59,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1680861299162, max timestamp=1680861299166Num uniques 1093; 2023-04-07 09:54:59,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1680861299162, max timestamp=1680861299166Num uniques 1093; 2023-04-07 09:54:59,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50048, new segment data size=11118, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-07 09:54:59,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50048, new segment data size=11118, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-07 09:54:59,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1680861299179, max timestamp=1680861299181Num uniques 1102; 2023-04-07 09:54:59,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-07 09:54:59,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1680861299179, max timestamp=1680861299181Num uniques 1102; 2023-04-07 09:54:59,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1680861299179, max timestamp=1680861299181Num uniques 1102; 2023-04-07 09:54:59,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-07 09:54:59,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-07 09:54:59,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680861299185, max timestamp=1680861299194Num uniques 1011; 2023-04-07 09:54:59,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-04-07 09:54:59,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680861299185, max timestamp=1680861299194Num uniques 1011; 2023-04-07 09:54:59,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-04-07 09:54:59,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1680861299185, max timestamp=1680861299194Num uniques 1011; 2023-04-07 09:54:59,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=9656, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-04-07 09:54:59,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1680861299203, max timestamp=1680861299208Num uniques 1070; 2023-04-07 09:54:59,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-04-07 09:54:59,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1680861299203, max timestamp=1680861299208Num uniques 1070; 2023-04-07 09:54:59,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-04-07 09:54:59,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1680861299203, max timestamp=1680861299208Num uniques 1070; 2023-04-07 09:54:59,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-04-07 09:54:59,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=279, cellSize=12774, totalHeapSize=29256, min timestamp=1680861299217, max timestamp=1680861299221Num uniques 1079; 2023-04-07 09:54:59,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=12774, suffix heap size=140184,new segment heap size=29256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=279 2023-04-07 09:54:59,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=279, cellSize=12774, totalHeapSize=29256, min timestamp=1680861299217, max timestamp=1680861299221Num uniques 1079; 2023-04-07 09:54:59,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=12774, suffix heap size=140184,new segment heap size=29256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=279 2023-04-07 09:54:59,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=279, cellSize=12774, totalHeapSize=29256, min timestamp=1680861299217, max timestamp=1680861299221Num uniques 1079; 2023-04-07 09:54:59,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=12774, suffix heap size=140184,new segment heap size=29256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=279 2023-04-07 09:54:59,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8598, totalHeapSize=19792, min timestamp=1680861299236, max timestamp=1680861299240Num uniques 1138; 2023-04-07 09:54:59,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52108, new segment data size=8598, suffix heap size=146320,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1138, new segment cells count=188 2023-04-07 09:54:59,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8598, totalHeapSize=19792, min timestamp=1680861299236, max timestamp=1680861299240Num uniques 1138; 2023-04-07 09:54:59,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52108, new segment data size=8598, suffix heap size=146320,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1138, new segment cells count=188 2023-04-07 09:54:59,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8599, totalHeapSize=19792, min timestamp=1680861299236, max timestamp=1680861299240Num uniques 1138; 2023-04-07 09:54:59,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52109, new segment data size=8599, suffix heap size=146320,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1138, new segment cells count=188 2023-04-07 09:54:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680861299248, max timestamp=1680861299257Num uniques 1047; 2023-04-07 09:54:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47944, new segment data size=7230, suffix heap size=136856,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=158 2023-04-07 09:54:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9012, totalHeapSize=20728, min timestamp=1680861299248, max timestamp=1680861299257Num uniques 1047; 2023-04-07 09:54:59,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47942, new segment data size=9012, suffix heap size=136856,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=197 2023-04-07 09:54:59,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1680861299248, max timestamp=1680861299257Num uniques 1047; 2023-04-07 09:54:59,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=9022, suffix heap size=136856,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=197 2023-04-07 09:54:59,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=256, cellSize=11720, totalHeapSize=26864, min timestamp=1680861299267, max timestamp=1680861299271Num uniques 1017; 2023-04-07 09:54:59,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46576, new segment data size=11720, suffix heap size=133736,new segment heap size=26864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=256 2023-04-07 09:54:59,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=256, cellSize=11720, totalHeapSize=26864, min timestamp=1680861299267, max timestamp=1680861299271Num uniques 1056; 2023-04-07 09:54:59,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=256, cellSize=11720, totalHeapSize=26864, min timestamp=1680861299267, max timestamp=1680861299271Num uniques 1056; 2023-04-07 09:54:59,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=11720, suffix heap size=137792,new segment heap size=26864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=256 2023-04-07 09:54:59,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=11720, suffix heap size=137792,new segment heap size=26864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=256 2023-04-07 09:54:59,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1680861299279, max timestamp=1680861299287Num uniques 1115; 2023-04-07 09:54:59,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51060, new segment data size=12130, suffix heap size=143928,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1115, new segment cells count=265 2023-04-07 09:54:59,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1680861299279, max timestamp=1680861299287Num uniques 1115; 2023-04-07 09:54:59,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51060, new segment data size=12130, suffix heap size=143928,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1115, new segment cells count=265 2023-04-07 09:54:59,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=265, cellSize=12130, totalHeapSize=27800, min timestamp=1680861299279, max timestamp=1680861299287Num uniques 1115; 2023-04-07 09:54:59,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51060, new segment data size=12130, suffix heap size=143928,new segment heap size=27800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1115, new segment cells count=265 2023-04-07 09:54:59,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7954, totalHeapSize=18336, min timestamp=1680861299296, max timestamp=1680861299301Num uniques 1124; 2023-04-07 09:54:59,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51464, new segment data size=7954, suffix heap size=144864,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=174 2023-04-07 09:54:59,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7960, totalHeapSize=18336, min timestamp=1680861299296, max timestamp=1680861299301Num uniques 1124; 2023-04-07 09:54:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51470, new segment data size=7960, suffix heap size=144864,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=174 2023-04-07 09:54:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=174, cellSize=7963, totalHeapSize=18336, min timestamp=1680861299296, max timestamp=1680861299301Num uniques 1124; 2023-04-07 09:54:59,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51473, new segment data size=7963, suffix heap size=144864,new segment heap size=18336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1124, new segment cells count=174 2023-04-07 09:54:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8361, totalHeapSize=19272, min timestamp=1680861299310, max timestamp=1680861299318Num uniques 1033; 2023-04-07 09:54:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47291, new segment data size=8361, suffix heap size=135400,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=183 2023-04-07 09:54:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861299310, max timestamp=1680861299318Num uniques 1033; 2023-04-07 09:54:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8366, totalHeapSize=19272, min timestamp=1680861299310, max timestamp=1680861299318Num uniques 1033; 2023-04-07 09:54:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47304, new segment data size=7050, suffix heap size=135400,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=154 2023-04-07 09:54:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47296, new segment data size=8366, suffix heap size=135400,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1033, new segment cells count=183 2023-04-07 09:54:59,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1680861299347, max timestamp=1680861299351Num uniques 1042; 2023-04-07 09:54:59,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=13372, suffix heap size=136336,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=292 2023-04-07 09:54:59,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1680861299348, max timestamp=1680861299351Num uniques 1013; 2023-04-07 09:54:59,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46398, new segment data size=8792, suffix heap size=133320,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=192 2023-04-07 09:54:59,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1680861299348, max timestamp=1680861299351Num uniques 1042; 2023-04-07 09:54:59,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47722, new segment data size=8792, suffix heap size=136336,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=192 2023-04-07 09:54:59,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=201, cellSize=9205, totalHeapSize=21144, min timestamp=1680861299361, max timestamp=1680861299365Num uniques 1051; 2023-04-07 09:54:59,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=201, cellSize=9205, totalHeapSize=21144, min timestamp=1680861299361, max timestamp=1680861299365Num uniques 1151; 2023-04-07 09:54:59,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52715, new segment data size=9205, suffix heap size=147672,new segment heap size=21144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=201 2023-04-07 09:54:59,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=9205, suffix heap size=137272,new segment heap size=21144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=201 2023-04-07 09:54:59,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=201, cellSize=9205, totalHeapSize=21144, min timestamp=1680861299361, max timestamp=1680861299365Num uniques 1051; 2023-04-07 09:54:59,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=9205, suffix heap size=137272,new segment heap size=21144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=201 2023-04-07 09:54:59,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1680861299373, max timestamp=1680861299379Num uniques 1060; 2023-04-07 09:54:59,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48540, new segment data size=9610, suffix heap size=138208,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1060, new segment cells count=210 2023-04-07 09:54:59,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1680861299373, max timestamp=1680861299379Num uniques 1060; 2023-04-07 09:54:59,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48540, new segment data size=9610, suffix heap size=138208,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1060, new segment cells count=210 2023-04-07 09:54:59,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1680861299373, max timestamp=1680861299379Num uniques 1060; 2023-04-07 09:54:59,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48540, new segment data size=9610, suffix heap size=138208,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1060, new segment cells count=210 2023-04-07 09:54:59,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1680861299392, max timestamp=1680861299396Num uniques 1069; 2023-04-07 09:54:59,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=12314, suffix heap size=139144,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=269 2023-04-07 09:54:59,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1680861299392, max timestamp=1680861299396Num uniques 1069; 2023-04-07 09:54:59,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=269, cellSize=12314, totalHeapSize=28216, min timestamp=1680861299392, max timestamp=1680861299396Num uniques 1069; 2023-04-07 09:54:59,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=12314, suffix heap size=139144,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=269 2023-04-07 09:54:59,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=12314, suffix heap size=139144,new segment heap size=28216 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=269 2023-04-07 09:54:59,407 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/aa3e9b1e62424b699fedd65a238187f6 2023-04-07 09:54:59,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1680861299399, max timestamp=1680861299410Num uniques 1128; 2023-04-07 09:54:59,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1680861299399, max timestamp=1680861299410Num uniques 1128; 2023-04-07 09:54:59,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-07 09:54:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-07 09:54:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1680861299399, max timestamp=1680861299410Num uniques 1128; 2023-04-07 09:54:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51658, new segment data size=10438, suffix heap size=145280,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1128, new segment cells count=228 2023-04-07 09:54:59,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1680861299421, max timestamp=1680861299424Num uniques 1087; 2023-04-07 09:54:59,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=10852, suffix heap size=141016,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=237 2023-04-07 09:54:59,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1680861299421, max timestamp=1680861299424Num uniques 1087; 2023-04-07 09:54:59,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=10852, suffix heap size=141016,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=237 2023-04-07 09:54:59,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1680861299421, max timestamp=1680861299424Num uniques 1087; 2023-04-07 09:54:59,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=10852, suffix heap size=141016,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=237 2023-04-07 09:54:59,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861299436, max timestamp=1680861299439Num uniques 1096; 2023-04-07 09:54:59,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=6870, suffix heap size=141952,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=150 2023-04-07 09:54:59,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=296, cellSize=13556, totalHeapSize=31024, min timestamp=1680861299435, max timestamp=1680861299439Num uniques 1096; 2023-04-07 09:54:59,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=13556, suffix heap size=141952,new segment heap size=31024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=296 2023-04-07 09:54:59,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=296, cellSize=13556, totalHeapSize=31024, min timestamp=1680861299435, max timestamp=1680861299439Num uniques 1096; 2023-04-07 09:54:59,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=13556, suffix heap size=141952,new segment heap size=31024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=296 2023-04-07 09:54:59,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1680861299453, max timestamp=1680861299459Num uniques 1155; 2023-04-07 09:54:59,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52895, new segment data size=7095, suffix heap size=148088,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1155, new segment cells count=155 2023-04-07 09:54:59,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861299453, max timestamp=1680861299460Num uniques 1155; 2023-04-07 09:54:59,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52895, new segment data size=6870, suffix heap size=148088,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1155, new segment cells count=150 2023-04-07 09:54:59,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861299453, max timestamp=1680861299460Num uniques 1009; 2023-04-07 09:54:59,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46209, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-07 09:54:59,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1680861299472, max timestamp=1680861299478Num uniques 1014; 2023-04-07 09:54:59,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=9794, suffix heap size=133424,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=214 2023-04-07 09:54:59,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1680861299472, max timestamp=1680861299478Num uniques 1009; 2023-04-07 09:54:59,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1680861299472, max timestamp=1680861299478Num uniques 1009; 2023-04-07 09:54:59,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46209, new segment data size=9794, suffix heap size=132904,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=214 2023-04-07 09:54:59,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46209, new segment data size=9794, suffix heap size=132904,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=214 2023-04-07 09:54:59,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7918, totalHeapSize=18232, min timestamp=1680861299490, max timestamp=1680861299494Num uniques 1073; 2023-04-07 09:54:59,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=7918, suffix heap size=139560,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=173 2023-04-07 09:54:59,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861299490, max timestamp=1680861299494Num uniques 1073; 2023-04-07 09:54:59,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=6870, suffix heap size=139560,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=150 2023-04-07 09:54:59,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861299490, max timestamp=1680861299494Num uniques 1073; 2023-04-07 09:54:59,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=6870, suffix heap size=139560,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=150 2023-04-07 09:54:59,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=182, cellSize=8332, totalHeapSize=19168, min timestamp=1680861299522, max timestamp=1680861299529Num uniques 1032; 2023-04-07 09:54:59,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=8332, suffix heap size=135296,new segment heap size=19168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=182 2023-04-07 09:54:59,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=182, cellSize=8332, totalHeapSize=19168, min timestamp=1680861299522, max timestamp=1680861299529Num uniques 1009; 2023-04-07 09:54:59,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=182, cellSize=8332, totalHeapSize=19168, min timestamp=1680861299522, max timestamp=1680861299529Num uniques 1009; 2023-04-07 09:54:59,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8332, suffix heap size=132904,new segment heap size=19168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=182 2023-04-07 09:54:59,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8332, suffix heap size=132904,new segment heap size=19168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=182 2023-04-07 09:54:59,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=291, cellSize=13326, totalHeapSize=30504, min timestamp=1680861299538, max timestamp=1680861299543Num uniques 1041; 2023-04-07 09:54:59,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47676, new segment data size=13326, suffix heap size=136232,new segment heap size=30504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1041, new segment cells count=291 2023-04-07 09:54:59,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=191, cellSize=8746, totalHeapSize=20104, min timestamp=1680861299538, max timestamp=1680861299543Num uniques 1041; 2023-04-07 09:54:59,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47676, new segment data size=8746, suffix heap size=136232,new segment heap size=20104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1041, new segment cells count=191 2023-04-07 09:54:59,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=191, cellSize=8746, totalHeapSize=20104, min timestamp=1680861299538, max timestamp=1680861299543Num uniques 1041; 2023-04-07 09:54:59,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47676, new segment data size=8746, suffix heap size=136232,new segment heap size=20104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1041, new segment cells count=191 2023-04-07 09:54:59,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1680861299552, max timestamp=1680861299560Num uniques 1150; 2023-04-07 09:54:59,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52670, new segment data size=11450, suffix heap size=147568,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1150, new segment cells count=250 2023-04-07 09:54:59,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1680861299552, max timestamp=1680861299560Num uniques 1050; 2023-04-07 09:54:59,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11440, totalHeapSize=26240, min timestamp=1680861299552, max timestamp=1680861299560Num uniques 1050; 2023-04-07 09:54:59,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48090, new segment data size=11450, suffix heap size=137168,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1050, new segment cells count=250 2023-04-07 09:54:59,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48080, new segment data size=11440, suffix heap size=137168,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1050, new segment cells count=250 2023-04-07 09:54:59,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861299570, max timestamp=1680861299599Num uniques 1109; 2023-04-07 09:54:59,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=7275, suffix heap size=143304,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=159 2023-04-07 09:54:59,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861299570, max timestamp=1680861299599Num uniques 1109; 2023-04-07 09:54:59,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861299570, max timestamp=1680861299599Num uniques 1109; 2023-04-07 09:54:59,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=7275, suffix heap size=143304,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=159 2023-04-07 09:54:59,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=7275, suffix heap size=143304,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=159 2023-04-07 09:54:59,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861299611, max timestamp=1680861299619Num uniques 1018; 2023-04-07 09:54:59,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:59,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861299611, max timestamp=1680861299619Num uniques 1018; 2023-04-07 09:54:59,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861299611, max timestamp=1680861299619Num uniques 1018; 2023-04-07 09:54:59,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:59,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:54:59,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861299632, max timestamp=1680861299639Num uniques 1027; 2023-04-07 09:54:59,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:59,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680861299632, max timestamp=1680861299639Num uniques 1027; 2023-04-07 09:54:59,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-07 09:54:59,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680861299632, max timestamp=1680861299639Num uniques 1027; 2023-04-07 09:54:59,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 09:54:59,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861299651, max timestamp=1680861299659Num uniques 1086; 2023-04-07 09:54:59,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-07 09:54:59,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861299651, max timestamp=1680861299659Num uniques 1086; 2023-04-07 09:54:59,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-07 09:54:59,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861299651, max timestamp=1680861299659Num uniques 1036; 2023-04-07 09:54:59,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-07 09:54:59,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861299669, max timestamp=1680861299675Num uniques 1095; 2023-04-07 09:54:59,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-07 09:54:59,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861299669, max timestamp=1680861299675Num uniques 1095; 2023-04-07 09:54:59,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-07 09:54:59,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861299669, max timestamp=1680861299675Num uniques 1095; 2023-04-07 09:54:59,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-07 09:54:59,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861299685, max timestamp=1680861299692Num uniques 1154; 2023-04-07 09:54:59,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-07 09:54:59,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861299685, max timestamp=1680861299692Num uniques 1154; 2023-04-07 09:54:59,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-07 09:54:59,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680861299685, max timestamp=1680861299692Num uniques 1154; 2023-04-07 09:54:59,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-07 09:54:59,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680861299699, max timestamp=1680861299705Num uniques 1013; 2023-04-07 09:54:59,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-07 09:54:59,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680861299699, max timestamp=1680861299705Num uniques 1013; 2023-04-07 09:54:59,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680861299699, max timestamp=1680861299705Num uniques 1013; 2023-04-07 09:54:59,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-07 09:54:59,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-07 09:54:59,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680861299713, max timestamp=1680861299719Num uniques 1072; 2023-04-07 09:54:59,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-07 09:54:59,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680861299713, max timestamp=1680861299719Num uniques 1072; 2023-04-07 09:54:59,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-07 09:54:59,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680861299713, max timestamp=1680861299719Num uniques 1072; 2023-04-07 09:54:59,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-07 09:54:59,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680861299732, max timestamp=1680861299735Num uniques 1081; 2023-04-07 09:54:59,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680861299732, max timestamp=1680861299735Num uniques 1081; 2023-04-07 09:54:59,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680861299732, max timestamp=1680861299735Num uniques 1081; 2023-04-07 09:54:59,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-07 09:54:59,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-07 09:54:59,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-07 09:54:59,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680861299745, max timestamp=1680861299750Num uniques 1040; 2023-04-07 09:54:59,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-07 09:54:59,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680861299745, max timestamp=1680861299750Num uniques 1040; 2023-04-07 09:54:59,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-07 09:54:59,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680861299745, max timestamp=1680861299750Num uniques 1040; 2023-04-07 09:54:59,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-07 09:54:59,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680861299759, max timestamp=1680861299765Num uniques 1099; 2023-04-07 09:54:59,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-07 09:54:59,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680861299759, max timestamp=1680861299765Num uniques 1099; 2023-04-07 09:54:59,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-07 09:54:59,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680861299759, max timestamp=1680861299765Num uniques 1099; 2023-04-07 09:54:59,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-07 09:54:59,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680861299774, max timestamp=1680861299782Num uniques 1158; 2023-04-07 09:54:59,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-07 09:54:59,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680861299774, max timestamp=1680861299782Num uniques 1158; 2023-04-07 09:54:59,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-07 09:54:59,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680861299774, max timestamp=1680861299782Num uniques 1158; 2023-04-07 09:54:59,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-07 09:54:59,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680861299793, max timestamp=1680861299798Num uniques 1117; 2023-04-07 09:54:59,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-07 09:54:59,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680861299793, max timestamp=1680861299798Num uniques 1117; 2023-04-07 09:54:59,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-07 09:54:59,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680861299793, max timestamp=1680861299798Num uniques 1117; 2023-04-07 09:54:59,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-04-07 09:54:59,817 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/66265b1134b84714aade287cc968f5ba 2023-04-07 09:54:59,820 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/a1b60be6e9a94c3db88065a912e59ed8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a1b60be6e9a94c3db88065a912e59ed8 2023-04-07 09:54:59,823 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a1b60be6e9a94c3db88065a912e59ed8, entries=50, sequenceid=635, filesize=7.1 K 2023-04-07 09:54:59,824 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/aa3e9b1e62424b699fedd65a238187f6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/aa3e9b1e62424b699fedd65a238187f6 2023-04-07 09:54:59,827 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/aa3e9b1e62424b699fedd65a238187f6, entries=50, sequenceid=635, filesize=7.1 K 2023-04-07 09:54:59,828 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/66265b1134b84714aade287cc968f5ba as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/66265b1134b84714aade287cc968f5ba 2023-04-07 09:54:59,833 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/66265b1134b84714aade287cc968f5ba, entries=50, sequenceid=635, filesize=7.1 K 2023-04-07 09:54:59,834 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=120.76 KB/123660 for 243bf66836bfdaa40ee21a1c79eaec90 in 1256ms, sequenceid=635, compaction requested=true 2023-04-07 09:54:59,834 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:54:59,834 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:59,834 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:59,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:59,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:54:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:59,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=127.47 KB heapSize=364.31 KB 2023-04-07 09:54:59,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:59,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17641304[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:59,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:59,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:54:59,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:59,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:54:59,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:59,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:54:59,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:54:59,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:59,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:59,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:59,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:59,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:59,846 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:59,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:59,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:59,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:59,847 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:59,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861359841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,848 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,848 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:59,848 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:59,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:59,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:59,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861359845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861359847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861359848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861359848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:59,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:59,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:59,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:59,853 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:59,853 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:54:59,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861359948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,953 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:54:59,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:54:59,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861359952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861359952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:54:59,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861359952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:54:59,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861359953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:54:59,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:54:59,956 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:54:59,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:59,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:54:59,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:54:59,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:59,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:59,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:59,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:54:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:59,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:54:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:59,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:54:59,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:54:59,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:54:59,959 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:59,959 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:54:59,959 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:54:59,959 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:54:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:54:59,960 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:54:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:59,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:54:59,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a117c84[Count = 0] remaining members to acquire global barrier 2023-04-07 09:54:59,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:54:59,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:54:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:54:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:54:59,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:54:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:54:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:59,966 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,966 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:54:59,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:54:59,968 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:54:59,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:54:59,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:54:59,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:54:59,968 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:54:59,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:54:59,970 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,970 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:54:59,971 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:54:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:54:59,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:54:59,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:54:59,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:54:59,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:54:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:54:59,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:54:59,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:54:59,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:54:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:54:59,974 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:54:59,974 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,074 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,078 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,079 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,080 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,080 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,086 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,086 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,086 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,087 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,089 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67b3119f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,093 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,100 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,100 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:00,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,115 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,115 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861360152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861360155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861360155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861360156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861360157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,216 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,219 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,224 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d98949d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,229 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,229 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,229 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,231 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,231 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,232 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,232 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,234 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:00,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,237 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,237 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1741 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/54e87fbac60f46c8a53cc4bae199101e 2023-04-07 09:55:00,337 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,340 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,343 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,346 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad35819[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,347 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,347 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,349 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,349 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,349 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,350 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,352 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,352 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,355 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,355 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,355 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:00,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,359 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,359 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861360457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,459 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,462 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861360458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861360459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861360461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861360461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,464 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,466 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,466 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,466 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,466 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bc7e49b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,472 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,474 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,474 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,484 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,484 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,584 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,587 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,593 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34873b95[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,598 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,600 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,603 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,603 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,604 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,604 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,604 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:00,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,610 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,610 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1741 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/1496230e4dfe4112805c0bd9d888500c 2023-04-07 09:55:00,710 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,713 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,715 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,715 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,715 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,715 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,719 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d627de4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,722 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,722 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,723 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,723 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,723 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,726 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,726 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,727 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,727 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,728 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,732 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,733 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:00,733 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,833 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,836 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,838 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d41e3c1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,851 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,854 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,854 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,856 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:00,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,859 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,859 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1680861360958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,960 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:00,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:00,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:00,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:00,963 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:00,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:00,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:00,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48354 deadline: 1680861360964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,965 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,966 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:00,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:00,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:00,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:00,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:00,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,968 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,968 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:00,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48332 deadline: 1680861360966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,968 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:00,969 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:00,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48318 deadline: 1680861360967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:00,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48308 deadline: 1680861360967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,972 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:00,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d627ce[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:00,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:00,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,975 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:00,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:00,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:00,975 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:00,977 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:00,980 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:00,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:00,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:00,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:00,980 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:00,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:00,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:00,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:00,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:00,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:00,990 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:00,990 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:00,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:00,991 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:00,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:01,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1741 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/7a67d27381d24a7890b10034a32f3d28 2023-04-07 09:55:01,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/54e87fbac60f46c8a53cc4bae199101e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/54e87fbac60f46c8a53cc4bae199101e 2023-04-07 09:55:01,091 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:01,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:01,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:01,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/54e87fbac60f46c8a53cc4bae199101e, entries=200, sequenceid=1741, filesize=14.3 K 2023-04-07 09:55:01,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:01,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:01,096 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:01,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:01,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:01,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:01,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/1496230e4dfe4112805c0bd9d888500c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/1496230e4dfe4112805c0bd9d888500c 2023-04-07 09:55:01,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,098 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:01,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:01,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:01,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:01,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:01,108 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:01,108 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:01,109 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/1496230e4dfe4112805c0bd9d888500c, entries=150, sequenceid=1741, filesize=11.9 K 2023-04-07 09:55:01,109 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:01,109 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:01,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/7a67d27381d24a7890b10034a32f3d28 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/7a67d27381d24a7890b10034a32f3d28 2023-04-07 09:55:01,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:01,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:01,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2480faf2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:01,112 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:01,112 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/7a67d27381d24a7890b10034a32f3d28, entries=150, sequenceid=1741, filesize=11.9 K 2023-04-07 09:55:01,113 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:01,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:01,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~424.73 KB/434928, currentSize=33.54 KB/34350 for 243bf66836bfdaa40ee21a1c79eaec90 in 1276ms, sequenceid=1741, compaction requested=true 2023-04-07 09:55:01,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:55:01,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:01,114 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:01,114 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:01,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:01,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:01,116 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47524 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:01,116 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/A is initiating minor compaction (all files) 2023-04-07 09:55:01,116 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/A in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:01,116 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a8f00d344b2040f7ac8605877d582d83, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/e57ea5b483344f97928e12ed3fbe0331, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a1b60be6e9a94c3db88065a912e59ed8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/54e87fbac60f46c8a53cc4bae199101e] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=46.4 K 2023-04-07 09:55:01,117 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:01,117 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/B is initiating minor compaction (all files) 2023-04-07 09:55:01,117 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/B in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:01,117 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f967e2285fcb4346a3e7473e7300ec46, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9dadba4b93984f09a8786a38e7047d60, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/aa3e9b1e62424b699fedd65a238187f6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/1496230e4dfe4112805c0bd9d888500c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=44.0 K 2023-04-07 09:55:01,118 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting a8f00d344b2040f7ac8605877d582d83, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680861297124 2023-04-07 09:55:01,118 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e57ea5b483344f97928e12ed3fbe0331, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680861297436 2023-04-07 09:55:01,118 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f967e2285fcb4346a3e7473e7300ec46, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680861297124 2023-04-07 09:55:01,119 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting a1b60be6e9a94c3db88065a912e59ed8, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1680861298577 2023-04-07 09:55:01,119 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9dadba4b93984f09a8786a38e7047d60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680861297436 2023-04-07 09:55:01,119 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 54e87fbac60f46c8a53cc4bae199101e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1741, earliestPutTs=1680861299832 2023-04-07 09:55:01,119 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting aa3e9b1e62424b699fedd65a238187f6, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1680861298577 2023-04-07 09:55:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:01,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,119 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1496230e4dfe4112805c0bd9d888500c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1741, earliestPutTs=1680861299832 2023-04-07 09:55:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:01,122 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:01,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:01,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:01,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:01,123 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:01,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:01,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:01,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:01,126 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:01,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:01,128 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#B#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:01,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:01,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:01,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:01,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:01,131 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:01,131 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:01,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:01,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:01,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:01,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,133 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#A#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:01,231 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:01,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:01,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:01,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:01,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:01,235 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:01,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:01,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:01,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:01,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:01,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:01,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:01,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:01,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:01,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:01,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. started... 2023-04-07 09:55:01,238 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:55:01,238 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:01,423 DEBUG [Thread-1845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:49660 2023-04-07 09:55:01,423 DEBUG [Thread-1845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,424 DEBUG [Thread-1849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:49660 2023-04-07 09:55:01,424 DEBUG [Thread-1847] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:49660 2023-04-07 09:55:01,424 DEBUG [Thread-1849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,424 DEBUG [Thread-1847] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,424 DEBUG [Thread-1841] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:49660 2023-04-07 09:55:01,424 DEBUG [Thread-1841] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,425 DEBUG [Thread-1843] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:49660 2023-04-07 09:55:01,425 DEBUG [Thread-1843] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,556 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/fb161e1c86a1456baa95a866ca26f2d1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fb161e1c86a1456baa95a866ca26f2d1 2023-04-07 09:55:01,557 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/19d7d418223c4058b785107ec71feb60 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/19d7d418223c4058b785107ec71feb60 2023-04-07 09:55:01,560 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/B of 243bf66836bfdaa40ee21a1c79eaec90 into fb161e1c86a1456baa95a866ca26f2d1(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:01,560 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:55:01,560 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/B, priority=12, startTime=1680861301114; duration=0sec 2023-04-07 09:55:01,560 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:01,560 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:01,560 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/A of 243bf66836bfdaa40ee21a1c79eaec90 into 19d7d418223c4058b785107ec71feb60(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:01,560 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:55:01,560 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/A, priority=12, startTime=1680861301114; duration=0sec 2023-04-07 09:55:01,561 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:01,561 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:01,561 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 243bf66836bfdaa40ee21a1c79eaec90/C is initiating minor compaction (all files) 2023-04-07 09:55:01,561 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 243bf66836bfdaa40ee21a1c79eaec90/C in TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:01,561 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2ee889f3e12444ff88e93dc1a900a161, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/278fec4af0404d8c9d14a9ee8474c198, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/66265b1134b84714aade287cc968f5ba, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/7a67d27381d24a7890b10034a32f3d28] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp, totalSize=44.0 K 2023-04-07 09:55:01,561 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2ee889f3e12444ff88e93dc1a900a161, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=623, earliestPutTs=1680861297124 2023-04-07 09:55:01,562 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 278fec4af0404d8c9d14a9ee8474c198, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680861297436 2023-04-07 09:55:01,562 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 66265b1134b84714aade287cc968f5ba, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1680861298577 2023-04-07 09:55:01,562 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7a67d27381d24a7890b10034a32f3d28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1741, earliestPutTs=1680861299832 2023-04-07 09:55:01,567 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 243bf66836bfdaa40ee21a1c79eaec90#C#compaction#353 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:01,645 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1749 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/36215dd7b504416a917d46334944c293 2023-04-07 09:55:01,962 DEBUG [Thread-1834] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:49660 2023-04-07 09:55:01,962 DEBUG [Thread-1834] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:55:01,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. as already flushing 2023-04-07 09:55:01,970 DEBUG [Thread-1832] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:49660 2023-04-07 09:55:01,970 DEBUG [Thread-1832] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,971 DEBUG [Thread-1838] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:49660 2023-04-07 09:55:01,971 DEBUG [Thread-1838] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,974 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/119cb22d4bf542faba509067c31b34ad as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/119cb22d4bf542faba509067c31b34ad 2023-04-07 09:55:01,975 DEBUG [Thread-1836] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:49660 2023-04-07 09:55:01,975 DEBUG [Thread-1836] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,977 DEBUG [Thread-1830] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:49660 2023-04-07 09:55:01,977 DEBUG [Thread-1830] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:01,978 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 243bf66836bfdaa40ee21a1c79eaec90/C of 243bf66836bfdaa40ee21a1c79eaec90 into 119cb22d4bf542faba509067c31b34ad(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:01,978 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:55:01,978 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90., storeName=243bf66836bfdaa40ee21a1c79eaec90/C, priority=12, startTime=1680861301115; duration=0sec 2023-04-07 09:55:01,978 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:02,053 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1749 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/687d4686d7f5400d9b91711ba4779c03 2023-04-07 09:55:02,461 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1749 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/358438dfdcdf40b681291f40a00ffbf2 2023-04-07 09:55:02,464 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/36215dd7b504416a917d46334944c293 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/36215dd7b504416a917d46334944c293 2023-04-07 09:55:02,468 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/36215dd7b504416a917d46334944c293, entries=100, sequenceid=1749, filesize=9.5 K 2023-04-07 09:55:02,469 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/687d4686d7f5400d9b91711ba4779c03 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/687d4686d7f5400d9b91711ba4779c03 2023-04-07 09:55:02,472 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/687d4686d7f5400d9b91711ba4779c03, entries=100, sequenceid=1749, filesize=9.5 K 2023-04-07 09:55:02,473 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/358438dfdcdf40b681291f40a00ffbf2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/358438dfdcdf40b681291f40a00ffbf2 2023-04-07 09:55:02,476 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/358438dfdcdf40b681291f40a00ffbf2, entries=100, sequenceid=1749, filesize=9.5 K 2023-04-07 09:55:02,476 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=33.54 KB/34350 for 243bf66836bfdaa40ee21a1c79eaec90 in 1238ms, sequenceid=1749, compaction requested=false 2023-04-07 09:55:02,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:55:02,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:02,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,478 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:02,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:02,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:02,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af33edb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:02,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:02,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:02,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:02,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:02,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:02,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:02,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:02,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:02,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:02,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:02,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:02,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:02,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:02,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:02,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:02,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:02,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:02,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:02,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:02,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:02,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:02,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:02,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:02,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:02,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:02,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:02,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:02,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,493 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:02,493 DEBUG [Thread-1840] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:02,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,493 DEBUG [Thread-1840] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:02,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:02,494 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:02,594 DEBUG [Thread-1840] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:02,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:02,594 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 359 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 341 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 230 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 342 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 359 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4935 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 5144 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4862 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4942 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 5157 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 09:55:02,595 INFO [Listener at localhost/39511] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 09:55:02,595 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:49660 2023-04-07 09:55:02,595 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:02,596 INFO [Listener at localhost/39511] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 09:55:02,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 09:55:02,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:02,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 09:55:02,599 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861302599"}]},"ts":"1680861302599"} 2023-04-07 09:55:02,600 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 09:55:02,601 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 09:55:02,602 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=243bf66836bfdaa40ee21a1c79eaec90, UNASSIGN}] 2023-04-07 09:55:02,603 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=243bf66836bfdaa40ee21a1c79eaec90, UNASSIGN 2023-04-07 09:55:02,603 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=243bf66836bfdaa40ee21a1c79eaec90, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:02,604 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861302603"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861302603"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861302603"}]},"ts":"1680861302603"} 2023-04-07 09:55:02,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:02,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 09:55:02,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:55:02,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 243bf66836bfdaa40ee21a1c79eaec90, disabling compactions & flushes 2023-04-07 09:55:02,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:02,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:02,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. after waiting 0 ms 2023-04-07 09:55:02,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:02,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 243bf66836bfdaa40ee21a1c79eaec90 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 09:55:02,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=A 2023-04-07 09:55:02,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:02,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=B 2023-04-07 09:55:02,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:02,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 243bf66836bfdaa40ee21a1c79eaec90, store=C 2023-04-07 09:55:02,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:02,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 09:55:03,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1760 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/b781ea9d9b684b17ae30395dd50794b1 2023-04-07 09:55:03,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 09:55:03,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1760 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/f72b0d2bdc2f44c5a866a3314f256f67 2023-04-07 09:55:03,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 09:55:03,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1760 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/d258599a79b54c54988c9af3acc4ca44 2023-04-07 09:55:03,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/A/b781ea9d9b684b17ae30395dd50794b1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b781ea9d9b684b17ae30395dd50794b1 2023-04-07 09:55:03,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b781ea9d9b684b17ae30395dd50794b1, entries=150, sequenceid=1760, filesize=11.9 K 2023-04-07 09:55:03,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/B/f72b0d2bdc2f44c5a866a3314f256f67 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f72b0d2bdc2f44c5a866a3314f256f67 2023-04-07 09:55:03,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f72b0d2bdc2f44c5a866a3314f256f67, entries=150, sequenceid=1760, filesize=11.9 K 2023-04-07 09:55:03,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/.tmp/C/d258599a79b54c54988c9af3acc4ca44 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/d258599a79b54c54988c9af3acc4ca44 2023-04-07 09:55:03,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/d258599a79b54c54988c9af3acc4ca44, entries=150, sequenceid=1760, filesize=11.9 K 2023-04-07 09:55:03,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 243bf66836bfdaa40ee21a1c79eaec90 in 1236ms, sequenceid=1760, compaction requested=true 2023-04-07 09:55:04,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/929ce8a91ca74e77ba347f5baf161a46, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/8d060c8b1e964e228477d57e7da6dbdb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/936aa60d358d423fbaf46f12fb68d5e9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3d9f2bf71d57407dac4253c92aea9a3f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/082da09d8fe344d189e06a0d09beacce, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3dcdc94da10a4e4dad3223e1791e50fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/14964efc15e948e9b02b4211709384b5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/61b77bdda583422fb060940c34683ca3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/07f3b8bf63b04f1ea0c73c4776d94486, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/7f82b399285c4f24bdd4399baed95cba, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/2c4e1ca273c842b5b530d7ce418e75fa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3e11c230cf484240934cc6393b72e547, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ca95465ff02e4c55a8afd5b22d295f53, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/f7c0bb1354b04d78aa826b434e3d49f9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/bf55f472792e4d21a2e0dc935b10aa1d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c031fb0ea654aa995c678a17f770f89, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/98fe87129f6b43d0b36a57a7c190f697, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c4363b38abd4653a4f7d9675c33973f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/83bc7e239c5747b7bace99256c5e6351, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ec5ac61e20dc4be58fa32a769e97fb84, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/71d55da6139f49c697c7b70789358a60, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/4aac5fdf15d24755b3c282753f9c460b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/793bcae508bb4be5b6b97c7847d89c7a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/268394d0e12a4ed8bfd793a6692ada77, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1194aed7ae764fa483b2aa3a18bbbdf5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a9f3c645576143e5804225257ab3a175, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/494a440356ab49e58faf036403b1c94e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a5bd4b48835e4a608bc0cff3983211d9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1c9b5299543a4502a552c4959aa43884, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b08bf74444904509bc04c2957f31fda6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/6b56cd8a2fa84fcb8ad0fd440468671e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1f76dd50852840ff8382e67c13af8c3d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/785d7240e34b4235b12a9cde0d3b834d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/c23968a358d4400a881d939dad673a6b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a8f00d344b2040f7ac8605877d582d83, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/0d2fc0f101204c5b80a82ed52f9c878c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/e57ea5b483344f97928e12ed3fbe0331, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a1b60be6e9a94c3db88065a912e59ed8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/54e87fbac60f46c8a53cc4bae199101e] to archive 2023-04-07 09:55:04,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:04,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/929ce8a91ca74e77ba347f5baf161a46 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/929ce8a91ca74e77ba347f5baf161a46 2023-04-07 09:55:04,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/8d060c8b1e964e228477d57e7da6dbdb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/8d060c8b1e964e228477d57e7da6dbdb 2023-04-07 09:55:04,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/936aa60d358d423fbaf46f12fb68d5e9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/936aa60d358d423fbaf46f12fb68d5e9 2023-04-07 09:55:04,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3d9f2bf71d57407dac4253c92aea9a3f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3d9f2bf71d57407dac4253c92aea9a3f 2023-04-07 09:55:04,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/082da09d8fe344d189e06a0d09beacce to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/082da09d8fe344d189e06a0d09beacce 2023-04-07 09:55:04,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3dcdc94da10a4e4dad3223e1791e50fd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3dcdc94da10a4e4dad3223e1791e50fd 2023-04-07 09:55:04,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/14964efc15e948e9b02b4211709384b5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/14964efc15e948e9b02b4211709384b5 2023-04-07 09:55:04,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/61b77bdda583422fb060940c34683ca3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/61b77bdda583422fb060940c34683ca3 2023-04-07 09:55:04,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/07f3b8bf63b04f1ea0c73c4776d94486 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/07f3b8bf63b04f1ea0c73c4776d94486 2023-04-07 09:55:04,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/7f82b399285c4f24bdd4399baed95cba to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/7f82b399285c4f24bdd4399baed95cba 2023-04-07 09:55:04,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/2c4e1ca273c842b5b530d7ce418e75fa to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/2c4e1ca273c842b5b530d7ce418e75fa 2023-04-07 09:55:04,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/9d8cbecbf3ad4b08be3a1ffe9f1f08ef 2023-04-07 09:55:04,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3e11c230cf484240934cc6393b72e547 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3e11c230cf484240934cc6393b72e547 2023-04-07 09:55:04,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ca95465ff02e4c55a8afd5b22d295f53 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ca95465ff02e4c55a8afd5b22d295f53 2023-04-07 09:55:04,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/f7c0bb1354b04d78aa826b434e3d49f9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/f7c0bb1354b04d78aa826b434e3d49f9 2023-04-07 09:55:04,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/bf55f472792e4d21a2e0dc935b10aa1d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/bf55f472792e4d21a2e0dc935b10aa1d 2023-04-07 09:55:04,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c031fb0ea654aa995c678a17f770f89 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c031fb0ea654aa995c678a17f770f89 2023-04-07 09:55:04,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/98fe87129f6b43d0b36a57a7c190f697 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/98fe87129f6b43d0b36a57a7c190f697 2023-04-07 09:55:04,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c4363b38abd4653a4f7d9675c33973f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/3c4363b38abd4653a4f7d9675c33973f 2023-04-07 09:55:04,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/83bc7e239c5747b7bace99256c5e6351 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/83bc7e239c5747b7bace99256c5e6351 2023-04-07 09:55:04,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ec5ac61e20dc4be58fa32a769e97fb84 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/ec5ac61e20dc4be58fa32a769e97fb84 2023-04-07 09:55:04,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/71d55da6139f49c697c7b70789358a60 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/71d55da6139f49c697c7b70789358a60 2023-04-07 09:55:04,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/4aac5fdf15d24755b3c282753f9c460b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/4aac5fdf15d24755b3c282753f9c460b 2023-04-07 09:55:04,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/793bcae508bb4be5b6b97c7847d89c7a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/793bcae508bb4be5b6b97c7847d89c7a 2023-04-07 09:55:04,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/268394d0e12a4ed8bfd793a6692ada77 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/268394d0e12a4ed8bfd793a6692ada77 2023-04-07 09:55:04,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1194aed7ae764fa483b2aa3a18bbbdf5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1194aed7ae764fa483b2aa3a18bbbdf5 2023-04-07 09:55:04,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a9f3c645576143e5804225257ab3a175 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a9f3c645576143e5804225257ab3a175 2023-04-07 09:55:04,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/494a440356ab49e58faf036403b1c94e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/494a440356ab49e58faf036403b1c94e 2023-04-07 09:55:04,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a5bd4b48835e4a608bc0cff3983211d9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a5bd4b48835e4a608bc0cff3983211d9 2023-04-07 09:55:04,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1c9b5299543a4502a552c4959aa43884 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1c9b5299543a4502a552c4959aa43884 2023-04-07 09:55:04,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b08bf74444904509bc04c2957f31fda6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b08bf74444904509bc04c2957f31fda6 2023-04-07 09:55:04,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/6b56cd8a2fa84fcb8ad0fd440468671e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/6b56cd8a2fa84fcb8ad0fd440468671e 2023-04-07 09:55:04,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1f76dd50852840ff8382e67c13af8c3d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/1f76dd50852840ff8382e67c13af8c3d 2023-04-07 09:55:04,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/785d7240e34b4235b12a9cde0d3b834d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/785d7240e34b4235b12a9cde0d3b834d 2023-04-07 09:55:04,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/c23968a358d4400a881d939dad673a6b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/c23968a358d4400a881d939dad673a6b 2023-04-07 09:55:04,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a8f00d344b2040f7ac8605877d582d83 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a8f00d344b2040f7ac8605877d582d83 2023-04-07 09:55:04,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/0d2fc0f101204c5b80a82ed52f9c878c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/0d2fc0f101204c5b80a82ed52f9c878c 2023-04-07 09:55:04,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/e57ea5b483344f97928e12ed3fbe0331 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/e57ea5b483344f97928e12ed3fbe0331 2023-04-07 09:55:04,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a1b60be6e9a94c3db88065a912e59ed8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/a1b60be6e9a94c3db88065a912e59ed8 2023-04-07 09:55:04,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/54e87fbac60f46c8a53cc4bae199101e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/54e87fbac60f46c8a53cc4bae199101e 2023-04-07 09:55:04,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a1dff585c7ec40c298bdb86ade929f31, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7ba8e5380515496f807b40fbfca9a653, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/78c37007227e478e9169673bb8b18e1a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e988fe4ab3a74b19b65d3daaae5526cd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2d39eaecbaa14c8d8c651e6a3cdaf5de, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4319a3905c54026ae42b6ca33d9faf9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7249f3ab34414c2b924c4898b11624fe, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7168cf55c50d48e88edfcc0b9bf81e82, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/09e79d7107ce4c33a89e01031baf9b66, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2fd6953f99f24ff783b4305e853e7152, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/27334d48face42cfa2ae4d503b26fe19, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/642572c56a7e4b419c9b401963158e87, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/94cfc709e6f94b629efe25422a1dee9b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/bcc0559fb2d2477e8a2b88cdbcd33efb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fe1ab0a1e07448c2a8adc7afd21a9977, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/c8f764706144414bb3fe0da7073d3ed3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d42b7053a28948559f00e7b286757683, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/313a0b48679e417e9bdad89f6423b960, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e138b8c8802341f0ab7ed33cf37b307d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2b23888fccdf4ca9b73708b970ae2557, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/dcf8516bf12d4e55991fdf7bb7a8a49a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2a069cb6fb5845a5b7b223bccdcf7242, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4f8019f653b4780bcdbc84d640f2529, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/4f0763b1f874420a8b4713ffd52a81b1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9070762c3f5544b2ba2a1c606ca58440, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/ef57bec06ecf4f6ba995468f81b6a69d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/8f8f881831434829b8a696022422aa48, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/212c9f654ca74f7c89c8a364d602e58c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/03aadf89477743b38fafe0ae9726b1b4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a69d1d5d6a31413e8e03d7c18b52fc07, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/108125c6e43f48c4a9300483b1918b37, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/520ad3ef3f0e40218b2a7dddf23f4a71, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/cb27048c940f435f997cb77cea704875, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d4f5720f6c1a44739057e406b48d639c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/db1df12d5fd645949685c32dcdbc6979, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f967e2285fcb4346a3e7473e7300ec46, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/6c2cae7a4d184d16b536e73349d4df86, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9dadba4b93984f09a8786a38e7047d60, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/aa3e9b1e62424b699fedd65a238187f6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/1496230e4dfe4112805c0bd9d888500c] to archive 2023-04-07 09:55:04,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:04,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a1dff585c7ec40c298bdb86ade929f31 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a1dff585c7ec40c298bdb86ade929f31 2023-04-07 09:55:04,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7ba8e5380515496f807b40fbfca9a653 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7ba8e5380515496f807b40fbfca9a653 2023-04-07 09:55:04,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/78c37007227e478e9169673bb8b18e1a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/78c37007227e478e9169673bb8b18e1a 2023-04-07 09:55:04,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e988fe4ab3a74b19b65d3daaae5526cd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e988fe4ab3a74b19b65d3daaae5526cd 2023-04-07 09:55:04,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2d39eaecbaa14c8d8c651e6a3cdaf5de to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2d39eaecbaa14c8d8c651e6a3cdaf5de 2023-04-07 09:55:04,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4319a3905c54026ae42b6ca33d9faf9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4319a3905c54026ae42b6ca33d9faf9 2023-04-07 09:55:04,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7249f3ab34414c2b924c4898b11624fe to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7249f3ab34414c2b924c4898b11624fe 2023-04-07 09:55:04,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7168cf55c50d48e88edfcc0b9bf81e82 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/7168cf55c50d48e88edfcc0b9bf81e82 2023-04-07 09:55:04,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/09e79d7107ce4c33a89e01031baf9b66 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/09e79d7107ce4c33a89e01031baf9b66 2023-04-07 09:55:04,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2fd6953f99f24ff783b4305e853e7152 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2fd6953f99f24ff783b4305e853e7152 2023-04-07 09:55:04,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/27334d48face42cfa2ae4d503b26fe19 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/27334d48face42cfa2ae4d503b26fe19 2023-04-07 09:55:04,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/642572c56a7e4b419c9b401963158e87 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/642572c56a7e4b419c9b401963158e87 2023-04-07 09:55:04,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/94cfc709e6f94b629efe25422a1dee9b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/94cfc709e6f94b629efe25422a1dee9b 2023-04-07 09:55:04,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/bcc0559fb2d2477e8a2b88cdbcd33efb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/bcc0559fb2d2477e8a2b88cdbcd33efb 2023-04-07 09:55:04,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fe1ab0a1e07448c2a8adc7afd21a9977 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fe1ab0a1e07448c2a8adc7afd21a9977 2023-04-07 09:55:04,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/c8f764706144414bb3fe0da7073d3ed3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/c8f764706144414bb3fe0da7073d3ed3 2023-04-07 09:55:04,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d42b7053a28948559f00e7b286757683 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d42b7053a28948559f00e7b286757683 2023-04-07 09:55:04,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/313a0b48679e417e9bdad89f6423b960 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/313a0b48679e417e9bdad89f6423b960 2023-04-07 09:55:04,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e138b8c8802341f0ab7ed33cf37b307d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/e138b8c8802341f0ab7ed33cf37b307d 2023-04-07 09:55:04,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2b23888fccdf4ca9b73708b970ae2557 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2b23888fccdf4ca9b73708b970ae2557 2023-04-07 09:55:04,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/dcf8516bf12d4e55991fdf7bb7a8a49a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/dcf8516bf12d4e55991fdf7bb7a8a49a 2023-04-07 09:55:04,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2a069cb6fb5845a5b7b223bccdcf7242 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/2a069cb6fb5845a5b7b223bccdcf7242 2023-04-07 09:55:04,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4f8019f653b4780bcdbc84d640f2529 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a4f8019f653b4780bcdbc84d640f2529 2023-04-07 09:55:04,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/4f0763b1f874420a8b4713ffd52a81b1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/4f0763b1f874420a8b4713ffd52a81b1 2023-04-07 09:55:04,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9070762c3f5544b2ba2a1c606ca58440 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9070762c3f5544b2ba2a1c606ca58440 2023-04-07 09:55:04,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/ef57bec06ecf4f6ba995468f81b6a69d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/ef57bec06ecf4f6ba995468f81b6a69d 2023-04-07 09:55:04,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/8f8f881831434829b8a696022422aa48 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/8f8f881831434829b8a696022422aa48 2023-04-07 09:55:04,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/212c9f654ca74f7c89c8a364d602e58c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/212c9f654ca74f7c89c8a364d602e58c 2023-04-07 09:55:04,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/03aadf89477743b38fafe0ae9726b1b4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/03aadf89477743b38fafe0ae9726b1b4 2023-04-07 09:55:04,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a69d1d5d6a31413e8e03d7c18b52fc07 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/a69d1d5d6a31413e8e03d7c18b52fc07 2023-04-07 09:55:04,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/108125c6e43f48c4a9300483b1918b37 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/108125c6e43f48c4a9300483b1918b37 2023-04-07 09:55:04,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/520ad3ef3f0e40218b2a7dddf23f4a71 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/520ad3ef3f0e40218b2a7dddf23f4a71 2023-04-07 09:55:04,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/cb27048c940f435f997cb77cea704875 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/cb27048c940f435f997cb77cea704875 2023-04-07 09:55:04,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d4f5720f6c1a44739057e406b48d639c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/d4f5720f6c1a44739057e406b48d639c 2023-04-07 09:55:04,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/db1df12d5fd645949685c32dcdbc6979 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/db1df12d5fd645949685c32dcdbc6979 2023-04-07 09:55:04,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f967e2285fcb4346a3e7473e7300ec46 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f967e2285fcb4346a3e7473e7300ec46 2023-04-07 09:55:04,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/6c2cae7a4d184d16b536e73349d4df86 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/6c2cae7a4d184d16b536e73349d4df86 2023-04-07 09:55:04,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9dadba4b93984f09a8786a38e7047d60 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/9dadba4b93984f09a8786a38e7047d60 2023-04-07 09:55:04,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/aa3e9b1e62424b699fedd65a238187f6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/aa3e9b1e62424b699fedd65a238187f6 2023-04-07 09:55:04,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/1496230e4dfe4112805c0bd9d888500c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/1496230e4dfe4112805c0bd9d888500c 2023-04-07 09:55:04,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/82864fc02fcc4e4e946f9ccdcc5d1cf4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/9811e6b56cc549369fd93275e061c26f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dfd51d4f8073414da2bd824c0884b2a2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3970929a211a489bb8e0d603f394846d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3d2fdcfe41894948b373a970d284aeee, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/13454b85e08d4f47a59356c4880ab546, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0ac1e066ab0841849e07893b244113c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3e29587626bb4850a11365d2bb335d08, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/64655276e60a4523951c2b12c4e17ff1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/b327d91d28644c95ae022e1f1d113d96, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e93aa737083246298432ff2691fb5d2c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/27977751b2e1447bab5ea6cf89873749, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/60c9c3b9e0df40eeb1a181e41038e4de, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/09249f30e3d84fdabf07b5a295c8313a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/8349bcd2629f4c3c85a424419a26f8eb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/ef260c40f49945ebb1b72717492892a9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e27a494f78774d1a9836352b57c60413, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/10adf5eb1e0c414ea3d26bc933ee82ea, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/4a527a21f4aa4f678e11ad224643c9c4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/15710071aec04ecab21356b5b848bf6d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/818ebbebb437420d9feadca675c179d3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/90e49a48f9b34933b1144940a7b1097b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/1ab37ffff02646d4abad5b2d5c736db7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/f832c62c6dd5405dabda80db113d28c9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2b08ab792e974a5a970c2545e83c4bfc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/426fa08d8f884e458bff6e73a7f76fa8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6c175fc5a1784e9aa1f80a276b762ed1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/68f8af648a174056bdac9c2a64d937dd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e37236f6056b479ca5c7ca461c343fb4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/a559748d7be546fea6c2005d2d44c1b0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/cdc73fae0c5840c7a1349c8eb3691bc5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6ec6375f32794f30a7e80f7783042b38, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/784b19451ea74ed98cfb7487840932c3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dbf120fb687d4ed2a74fc3d4f27e8bd3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2ee889f3e12444ff88e93dc1a900a161, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0f0751e62b6943d4a7b2add4eda4c516, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/278fec4af0404d8c9d14a9ee8474c198, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/66265b1134b84714aade287cc968f5ba, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/7a67d27381d24a7890b10034a32f3d28] to archive 2023-04-07 09:55:04,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:04,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/82864fc02fcc4e4e946f9ccdcc5d1cf4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/82864fc02fcc4e4e946f9ccdcc5d1cf4 2023-04-07 09:55:04,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/9811e6b56cc549369fd93275e061c26f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/9811e6b56cc549369fd93275e061c26f 2023-04-07 09:55:04,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dfd51d4f8073414da2bd824c0884b2a2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dfd51d4f8073414da2bd824c0884b2a2 2023-04-07 09:55:04,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3970929a211a489bb8e0d603f394846d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3970929a211a489bb8e0d603f394846d 2023-04-07 09:55:04,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3d2fdcfe41894948b373a970d284aeee to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3d2fdcfe41894948b373a970d284aeee 2023-04-07 09:55:04,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/13454b85e08d4f47a59356c4880ab546 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/13454b85e08d4f47a59356c4880ab546 2023-04-07 09:55:04,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0ac1e066ab0841849e07893b244113c1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0ac1e066ab0841849e07893b244113c1 2023-04-07 09:55:04,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3e29587626bb4850a11365d2bb335d08 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/3e29587626bb4850a11365d2bb335d08 2023-04-07 09:55:04,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/64655276e60a4523951c2b12c4e17ff1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/64655276e60a4523951c2b12c4e17ff1 2023-04-07 09:55:04,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/b327d91d28644c95ae022e1f1d113d96 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/b327d91d28644c95ae022e1f1d113d96 2023-04-07 09:55:04,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e93aa737083246298432ff2691fb5d2c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e93aa737083246298432ff2691fb5d2c 2023-04-07 09:55:04,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/27977751b2e1447bab5ea6cf89873749 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/27977751b2e1447bab5ea6cf89873749 2023-04-07 09:55:04,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/60c9c3b9e0df40eeb1a181e41038e4de to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/60c9c3b9e0df40eeb1a181e41038e4de 2023-04-07 09:55:04,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/09249f30e3d84fdabf07b5a295c8313a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/09249f30e3d84fdabf07b5a295c8313a 2023-04-07 09:55:04,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/8349bcd2629f4c3c85a424419a26f8eb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/8349bcd2629f4c3c85a424419a26f8eb 2023-04-07 09:55:04,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/ef260c40f49945ebb1b72717492892a9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/ef260c40f49945ebb1b72717492892a9 2023-04-07 09:55:04,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e27a494f78774d1a9836352b57c60413 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e27a494f78774d1a9836352b57c60413 2023-04-07 09:55:04,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/10adf5eb1e0c414ea3d26bc933ee82ea to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/10adf5eb1e0c414ea3d26bc933ee82ea 2023-04-07 09:55:04,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/4a527a21f4aa4f678e11ad224643c9c4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/4a527a21f4aa4f678e11ad224643c9c4 2023-04-07 09:55:04,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/15710071aec04ecab21356b5b848bf6d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/15710071aec04ecab21356b5b848bf6d 2023-04-07 09:55:04,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/818ebbebb437420d9feadca675c179d3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/818ebbebb437420d9feadca675c179d3 2023-04-07 09:55:04,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/90e49a48f9b34933b1144940a7b1097b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/90e49a48f9b34933b1144940a7b1097b 2023-04-07 09:55:04,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/1ab37ffff02646d4abad5b2d5c736db7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/1ab37ffff02646d4abad5b2d5c736db7 2023-04-07 09:55:04,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/f832c62c6dd5405dabda80db113d28c9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/f832c62c6dd5405dabda80db113d28c9 2023-04-07 09:55:04,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2b08ab792e974a5a970c2545e83c4bfc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2b08ab792e974a5a970c2545e83c4bfc 2023-04-07 09:55:04,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/426fa08d8f884e458bff6e73a7f76fa8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/426fa08d8f884e458bff6e73a7f76fa8 2023-04-07 09:55:04,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6c175fc5a1784e9aa1f80a276b762ed1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6c175fc5a1784e9aa1f80a276b762ed1 2023-04-07 09:55:04,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/68f8af648a174056bdac9c2a64d937dd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/68f8af648a174056bdac9c2a64d937dd 2023-04-07 09:55:04,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e37236f6056b479ca5c7ca461c343fb4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/e37236f6056b479ca5c7ca461c343fb4 2023-04-07 09:55:04,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/a559748d7be546fea6c2005d2d44c1b0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/a559748d7be546fea6c2005d2d44c1b0 2023-04-07 09:55:04,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/cdc73fae0c5840c7a1349c8eb3691bc5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/cdc73fae0c5840c7a1349c8eb3691bc5 2023-04-07 09:55:04,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6ec6375f32794f30a7e80f7783042b38 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/6ec6375f32794f30a7e80f7783042b38 2023-04-07 09:55:04,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/784b19451ea74ed98cfb7487840932c3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/784b19451ea74ed98cfb7487840932c3 2023-04-07 09:55:04,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dbf120fb687d4ed2a74fc3d4f27e8bd3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/dbf120fb687d4ed2a74fc3d4f27e8bd3 2023-04-07 09:55:04,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2ee889f3e12444ff88e93dc1a900a161 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/2ee889f3e12444ff88e93dc1a900a161 2023-04-07 09:55:04,115 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0f0751e62b6943d4a7b2add4eda4c516 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/0f0751e62b6943d4a7b2add4eda4c516 2023-04-07 09:55:04,116 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/278fec4af0404d8c9d14a9ee8474c198 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/278fec4af0404d8c9d14a9ee8474c198 2023-04-07 09:55:04,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/66265b1134b84714aade287cc968f5ba to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/66265b1134b84714aade287cc968f5ba 2023-04-07 09:55:04,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/7a67d27381d24a7890b10034a32f3d28 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/7a67d27381d24a7890b10034a32f3d28 2023-04-07 09:55:04,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/recovered.edits/1763.seqid, newMaxSeqId=1763, maxSeqId=1 2023-04-07 09:55:04,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90. 2023-04-07 09:55:04,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 243bf66836bfdaa40ee21a1c79eaec90: 2023-04-07 09:55:04,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:55:04,125 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=243bf66836bfdaa40ee21a1c79eaec90, regionState=CLOSED 2023-04-07 09:55:04,125 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861304125"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861304125"}]},"ts":"1680861304125"} 2023-04-07 09:55:04,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-07 09:55:04,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 243bf66836bfdaa40ee21a1c79eaec90, server=jenkins-hbase4.apache.org,35881,1680861218288 in 1.5220 sec 2023-04-07 09:55:04,127 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-07 09:55:04,127 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=243bf66836bfdaa40ee21a1c79eaec90, UNASSIGN in 1.5240 sec 2023-04-07 09:55:04,128 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861304128"}]},"ts":"1680861304128"} 2023-04-07 09:55:04,129 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 09:55:04,131 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 09:55:04,132 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5360 sec 2023-04-07 09:55:04,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-07 09:55:04,703 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-07 09:55:04,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 09:55:04,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:04,705 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:04,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-07 09:55:04,706 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:04,710 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:55:04,712 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/recovered.edits] 2023-04-07 09:55:04,714 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/19d7d418223c4058b785107ec71feb60 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/19d7d418223c4058b785107ec71feb60 2023-04-07 09:55:04,715 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/36215dd7b504416a917d46334944c293 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/36215dd7b504416a917d46334944c293 2023-04-07 09:55:04,716 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b781ea9d9b684b17ae30395dd50794b1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/A/b781ea9d9b684b17ae30395dd50794b1 2023-04-07 09:55:04,718 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/687d4686d7f5400d9b91711ba4779c03 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/687d4686d7f5400d9b91711ba4779c03 2023-04-07 09:55:04,719 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f72b0d2bdc2f44c5a866a3314f256f67 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/f72b0d2bdc2f44c5a866a3314f256f67 2023-04-07 09:55:04,720 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fb161e1c86a1456baa95a866ca26f2d1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/B/fb161e1c86a1456baa95a866ca26f2d1 2023-04-07 09:55:04,721 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/119cb22d4bf542faba509067c31b34ad to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/119cb22d4bf542faba509067c31b34ad 2023-04-07 09:55:04,722 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/358438dfdcdf40b681291f40a00ffbf2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/358438dfdcdf40b681291f40a00ffbf2 2023-04-07 09:55:04,723 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/d258599a79b54c54988c9af3acc4ca44 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/C/d258599a79b54c54988c9af3acc4ca44 2023-04-07 09:55:04,726 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/recovered.edits/1763.seqid to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90/recovered.edits/1763.seqid 2023-04-07 09:55:04,726 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/243bf66836bfdaa40ee21a1c79eaec90 2023-04-07 09:55:04,726 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:55:04,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:04,729 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 09:55:04,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 09:55:04,732 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:04,732 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 09:55:04,732 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680861304732"}]},"ts":"9223372036854775807"} 2023-04-07 09:55:04,733 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 09:55:04,733 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 243bf66836bfdaa40ee21a1c79eaec90, NAME => 'TestAcidGuarantees,,1680861279252.243bf66836bfdaa40ee21a1c79eaec90.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 09:55:04,733 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 09:55:04,733 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680861304733"}]},"ts":"9223372036854775807"} 2023-04-07 09:55:04,734 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 09:55:04,736 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:04,736 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-07 09:55:04,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-07 09:55:04,807 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-07 09:55:04,818 INFO [Listener at localhost/39511] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 233), OpenFileDescriptor=590 (was 558) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=486 (was 434) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=2712 (was 2810) 2023-04-07 09:55:04,830 INFO [Listener at localhost/39511] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=590, MaxFileDescriptor=60000, SystemLoadAverage=486, ProcessCount=169, AvailableMemoryMB=2712 2023-04-07 09:55:04,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:55:04,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:55:04,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:04,834 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 09:55:04,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-07 09:55:04,834 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 09:55:04,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 09:55:04,835 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:04,836 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c empty. 2023-04-07 09:55:04,836 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:04,836 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:55:04,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 09:55:05,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 09:55:05,242 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:55:05,244 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5aeda58872a183eaf0216fd608afed2c, NAME => 'TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp 2023-04-07 09:55:05,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 09:55:05,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:55:05,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5aeda58872a183eaf0216fd608afed2c, disabling compactions & flushes 2023-04-07 09:55:05,650 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. after waiting 0 ms 2023-04-07 09:55:05,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,650 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,650 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:05,652 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 09:55:05,653 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861305653"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861305653"}]},"ts":"1680861305653"} 2023-04-07 09:55:05,653 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 09:55:05,654 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 09:55:05,654 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861305654"}]},"ts":"1680861305654"} 2023-04-07 09:55:05,655 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 09:55:05,657 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, ASSIGN}] 2023-04-07 09:55:05,659 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, ASSIGN 2023-04-07 09:55:05,659 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=false 2023-04-07 09:55:05,810 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:05,810 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861305810"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861305810"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861305810"}]},"ts":"1680861305810"} 2023-04-07 09:55:05,811 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:05,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 09:55:05,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5aeda58872a183eaf0216fd608afed2c, NAME => 'TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:55:05,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:55:05,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,968 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,969 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A 2023-04-07 09:55:05,969 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A 2023-04-07 09:55:05,969 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:05,969 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5aeda58872a183eaf0216fd608afed2c columnFamilyName A 2023-04-07 09:55:05,970 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(310): Store=5aeda58872a183eaf0216fd608afed2c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:05,970 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,971 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B 2023-04-07 09:55:05,971 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B 2023-04-07 09:55:05,971 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:05,971 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5aeda58872a183eaf0216fd608afed2c columnFamilyName B 2023-04-07 09:55:05,971 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(310): Store=5aeda58872a183eaf0216fd608afed2c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:05,971 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,972 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C 2023-04-07 09:55:05,972 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C 2023-04-07 09:55:05,972 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:05,973 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5aeda58872a183eaf0216fd608afed2c columnFamilyName C 2023-04-07 09:55:05,973 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(310): Store=5aeda58872a183eaf0216fd608afed2c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:05,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:55:05,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:05,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:55:05,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5aeda58872a183eaf0216fd608afed2c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63328173, jitterRate=-0.05633668601512909}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:55:05,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:05,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., pid=37, masterSystemTime=1680861305963 2023-04-07 09:55:05,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:05,980 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:05,980 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861305980"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861305980"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861305980"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861305980"}]},"ts":"1680861305980"} 2023-04-07 09:55:05,982 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-07 09:55:05,982 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 in 170 msec 2023-04-07 09:55:05,983 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-07 09:55:05,983 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, ASSIGN in 325 msec 2023-04-07 09:55:05,983 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 09:55:05,983 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861305983"}]},"ts":"1680861305983"} 2023-04-07 09:55:05,984 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 09:55:05,986 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 09:55:05,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1540 sec 2023-04-07 09:55:06,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-07 09:55:06,939 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-07 09:55:06,940 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:06,944 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:06,945 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:06,947 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:06,948 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 09:55:06,949 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 09:55:06,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:55:06,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:55:06,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:07,360 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 09:55:07,361 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:55:07,361 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 09:55:07,363 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-07 09:55:07,365 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, REOPEN/MOVE}] 2023-04-07 09:55:07,366 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, REOPEN/MOVE 2023-04-07 09:55:07,366 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,367 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861307366"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861307366"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861307366"}]},"ts":"1680861307366"} 2023-04-07 09:55:07,367 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:07,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5aeda58872a183eaf0216fd608afed2c, disabling compactions & flushes 2023-04-07 09:55:07,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. after waiting 0 ms 2023-04-07 09:55:07,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-07 09:55:07,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:07,524 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5aeda58872a183eaf0216fd608afed2c to self. 2023-04-07 09:55:07,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,525 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=CLOSED 2023-04-07 09:55:07,525 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861307525"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861307525"}]},"ts":"1680861307525"} 2023-04-07 09:55:07,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-07 09:55:07,527 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 in 159 msec 2023-04-07 09:55:07,528 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=true 2023-04-07 09:55:07,678 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,678 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861307678"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861307678"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861307678"}]},"ts":"1680861307678"} 2023-04-07 09:55:07,679 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:07,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5aeda58872a183eaf0216fd608afed2c, NAME => 'TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:55:07,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:55:07,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,836 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,836 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A 2023-04-07 09:55:07,836 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A 2023-04-07 09:55:07,837 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:07,837 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5aeda58872a183eaf0216fd608afed2c columnFamilyName A 2023-04-07 09:55:07,838 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(310): Store=5aeda58872a183eaf0216fd608afed2c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:07,838 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,839 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B 2023-04-07 09:55:07,839 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B 2023-04-07 09:55:07,839 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:07,839 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5aeda58872a183eaf0216fd608afed2c columnFamilyName B 2023-04-07 09:55:07,840 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(310): Store=5aeda58872a183eaf0216fd608afed2c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:07,840 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,840 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C 2023-04-07 09:55:07,841 DEBUG [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C 2023-04-07 09:55:07,841 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:07,841 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5aeda58872a183eaf0216fd608afed2c columnFamilyName C 2023-04-07 09:55:07,841 INFO [StoreOpener-5aeda58872a183eaf0216fd608afed2c-1] regionserver.HStore(310): Store=5aeda58872a183eaf0216fd608afed2c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:07,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:55:07,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5aeda58872a183eaf0216fd608afed2c; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60340143, jitterRate=-0.1008618026971817}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:55:07,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:07,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., pid=42, masterSystemTime=1680861307831 2023-04-07 09:55:07,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,847 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,847 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861307847"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861307847"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861307847"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861307847"}]},"ts":"1680861307847"} 2023-04-07 09:55:07,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-07 09:55:07,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 in 169 msec 2023-04-07 09:55:07,850 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-07 09:55:07,850 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, REOPEN/MOVE in 484 msec 2023-04-07 09:55:07,852 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-07 09:55:07,852 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-07 09:55:07,853 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-04-07 09:55:07,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-07 09:55:07,854 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,860 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,860 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,863 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,864 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,867 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,867 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,871 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,871 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,875 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,875 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,879 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,879 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,882 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,882 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,887 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,888 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,892 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,892 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x159c5e0c to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:07,895 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1774338b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:07,904 DEBUG [hconnection-0x5d00afd9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,905 DEBUG [hconnection-0x17cace9d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,906 DEBUG [hconnection-0x599837d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,906 DEBUG [hconnection-0x77e27de4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,908 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,908 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,908 DEBUG [hconnection-0x69fddec6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,908 DEBUG [hconnection-0x430a2f73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,909 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45124, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,909 DEBUG [hconnection-0x38c09b34-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,910 DEBUG [hconnection-0x74bcd1c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:07,909 DEBUG [hconnection-0x7a9b9f6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,909 DEBUG [hconnection-0x60f9976f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:07,909 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,914 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,921 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,921 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,923 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:07,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:07,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:07,928 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:07,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:07,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:07,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:07,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,930 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:07,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:07,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:07,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:07,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:07,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:07,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:07,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:07,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:07,933 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,933 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:07,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:07,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:07,933 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:07,933 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:07,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:07,935 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:07,935 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:07,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:07,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:07,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:07,936 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,936 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:07,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:07,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87e81af[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:07,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:07,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,942 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:07,942 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:07,942 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,943 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:07,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:07,944 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:07,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:07,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:07,946 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:07,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:07,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:07,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:07,947 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:07,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:07,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:07,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861367948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:07,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:07,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:07,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861367947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861367948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25594, totalHeapSize=58376, min timestamp=1680861307941, max timestamp=1680861307948Num uniques 859; 2023-04-07 09:55:07,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=25594, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-04-07 09:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:07,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25594, totalHeapSize=58376, min timestamp=1680861307941, max timestamp=1680861307948Num uniques 859; 2023-04-07 09:55:07,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=25594, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-04-07 09:55:07,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25594, totalHeapSize=58376, min timestamp=1680861307941, max timestamp=1680861307948Num uniques 859; 2023-04-07 09:55:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:07,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=25594, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-04-07 09:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,951 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,952 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:07,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:07,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:07,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,954 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:07,954 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:07,954 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:07,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:07,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:07,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861367967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:07,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:07,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861367969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,054 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861368051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861368051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,061 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861368056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,064 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,064 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,064 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,064 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,072 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861368070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861368070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,074 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,074 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,074 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,074 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108e2f39[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,079 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,079 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,082 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,086 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,086 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,089 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,089 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,089 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:08,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,094 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:08,095 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:08,195 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,203 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,204 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,204 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,204 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,205 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,206 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,206 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,206 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,206 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,206 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337a00b0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,215 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,217 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,219 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,219 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,221 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,221 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,221 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,223 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:08,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:08,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:08,226 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:08,227 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:08,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861368255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861368256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861368263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861368274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861368275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,327 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,331 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,331 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,331 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,331 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,333 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,335 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,335 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,335 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,335 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,335 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bce5e2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,345 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,351 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,351 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:08,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,358 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:08,358 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,358 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,365 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407061ef14588224fc68bb41be403174341 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407061ef14588224fc68bb41be403174341 2023-04-07 09:55:08,372 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/235ad67f3bc84bf1bd0384927759354d 2023-04-07 09:55:08,458 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,462 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,464 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,465 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,465 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,465 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,465 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,468 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb371a3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,472 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,475 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,475 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:08,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:08,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:08,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,482 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:08,482 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861368563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861368566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861368572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861368577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,582 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:08,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861368578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,592 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,593 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,593 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,594 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,594 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,594 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,594 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da570eb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,606 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,606 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,608 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,608 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,608 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,613 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,613 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,616 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,616 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,619 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,623 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:08,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:08,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,623 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:08,624 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:08,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,724 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,727 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,728 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,730 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,730 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,731 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,731 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e39826f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,735 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,735 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,736 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,736 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,736 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,738 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,741 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,741 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,742 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,742 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,744 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,746 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,749 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:08,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:08,749 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,749 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,750 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:08,750 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:08,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,780 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/82d0ed8ac6c5466d9e40a9a43c11930d 2023-04-07 09:55:08,850 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,853 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,856 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,856 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,856 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,856 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19bdf5bf[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,860 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,861 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,865 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,865 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,868 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:08,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:08,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:08,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,872 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:08,872 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:08,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,872 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,972 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:08,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:08,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:08,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:08,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:08,975 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:08,975 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:08,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:08,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,977 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,977 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:08,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:08,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:08,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:08,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:08,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:08,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:08,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:08,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,982 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:08,983 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:08,983 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:08,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:08,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:08,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:08,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:08,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:08,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e94b142[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:08,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:08,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:08,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:08,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:08,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:08,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:08,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:08,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:08,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:08,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:08,991 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:08,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:08,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:08,991 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:08,991 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:08,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:08,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:08,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:08,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:09,002 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:09,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:09,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:09,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:09,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:09,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:09,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,003 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:09,003 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:09,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:09,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:09,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861369073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861369073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:09,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861369081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:09,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861369085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:09,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861369085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,103 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:09,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:09,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:09,106 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:09,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:09,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:09,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:09,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:09,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:09,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:09,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:09,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:09,109 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:09,109 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:09,109 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:09,109 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:09,110 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:09,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,111 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:09,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:09,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:09,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f0aa53[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:09,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:09,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:09,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:09,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:09,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:09,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:09,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:09,119 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:09,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:09,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:09,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:09,119 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:09,120 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,120 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,120 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:09,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:09,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,126 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:09,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:09,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:09,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:09,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:09,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:09,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:09,131 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,131 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:09,131 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:09,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:09,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/1b8afb2f849748bab28858b5d7e4f83f 2023-04-07 09:55:09,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/235ad67f3bc84bf1bd0384927759354d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/235ad67f3bc84bf1bd0384927759354d 2023-04-07 09:55:09,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/235ad67f3bc84bf1bd0384927759354d, entries=150, sequenceid=15, filesize=25.1 K 2023-04-07 09:55:09,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/82d0ed8ac6c5466d9e40a9a43c11930d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/82d0ed8ac6c5466d9e40a9a43c11930d 2023-04-07 09:55:09,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/82d0ed8ac6c5466d9e40a9a43c11930d, entries=150, sequenceid=15, filesize=11.6 K 2023-04-07 09:55:09,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/1b8afb2f849748bab28858b5d7e4f83f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1b8afb2f849748bab28858b5d7e4f83f 2023-04-07 09:55:09,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1b8afb2f849748bab28858b5d7e4f83f, entries=150, sequenceid=15, filesize=11.6 K 2023-04-07 09:55:09,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 5aeda58872a183eaf0216fd608afed2c in 1275ms, sequenceid=15, compaction requested=false 2023-04-07 09:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-07 09:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:09,231 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:09,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:09,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:09,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:09,235 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:09,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:09,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:09,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:09,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,237 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:09,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:09,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:09,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:09,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:09,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:09,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:09,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:09,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:09,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:09,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:09,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:09,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:09,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:09,239 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=140.89 KB heapSize=367.41 KB 2023-04-07 09:55:09,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:09,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:09,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:09,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:09,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:09,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:09,650 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 09:55:09,652 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407d36adbe039714199a205b4c4104f4090 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d36adbe039714199a205b4c4104f4090 2023-04-07 09:55:09,659 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=45, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/b704f6947afe4d9da81dc28959661359 2023-04-07 09:55:09,681 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/12c5a4b9824f4e01ae8b0c9bc39d335b 2023-04-07 09:55:10,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:10,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,091 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/a8d2bb7d9ecd4d7f89c2455250eef463 2023-04-07 09:55:10,095 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/b704f6947afe4d9da81dc28959661359 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/b704f6947afe4d9da81dc28959661359 2023-04-07 09:55:10,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861370093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861370094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,101 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/b704f6947afe4d9da81dc28959661359, entries=150, sequenceid=45, filesize=25.1 K 2023-04-07 09:55:10,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861370094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,102 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/12c5a4b9824f4e01ae8b0c9bc39d335b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/12c5a4b9824f4e01ae8b0c9bc39d335b 2023-04-07 09:55:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861370095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861370096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,105 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/12c5a4b9824f4e01ae8b0c9bc39d335b, entries=150, sequenceid=45, filesize=11.6 K 2023-04-07 09:55:10,106 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/a8d2bb7d9ecd4d7f89c2455250eef463 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a8d2bb7d9ecd4d7f89c2455250eef463 2023-04-07 09:55:10,109 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a8d2bb7d9ecd4d7f89c2455250eef463, entries=150, sequenceid=45, filesize=11.6 K 2023-04-07 09:55:10,109 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~367.36 KB/376176, currentSize=53.67 KB/54960 for 5aeda58872a183eaf0216fd608afed2c in 870ms, sequenceid=45, compaction requested=false 2023-04-07 09:55:10,110 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:10,110 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,111 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9863d8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,114 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,114 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,118 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,118 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,122 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,122 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,122 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,122 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:10,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:10,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,125 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,125 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,126 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:10,126 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:10,126 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,126 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,126 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,126 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,126 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:10,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:10,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:55:10,200 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:10,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:10,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:10,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:10,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:10,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:10,226 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:10,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:10,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861370223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861370227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:10,230 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:10,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:10,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:10,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,232 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,232 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:10,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:10,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:10,233 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,233 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:10,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,233 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861370228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,233 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861370229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,235 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,235 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfe6f39[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,238 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,243 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,243 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:10,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:10,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,251 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:10,251 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,252 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:10,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861370330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861370330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861370335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861370336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,351 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:10,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:10,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:10,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:10,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:10,354 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:10,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:10,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:10,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,357 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,357 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:10,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:10,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:10,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:10,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:10,358 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:10,358 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,358 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:10,359 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,359 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57cbcf45[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,363 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,363 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,363 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,363 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,366 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,366 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,368 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,368 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,368 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,373 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,373 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,373 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,373 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,373 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:10,373 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,374 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,374 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,374 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:10,374 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:10,374 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,374 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,474 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:10,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:10,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:10,477 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:10,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:10,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:10,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,479 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:10,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:10,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:10,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:10,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:10,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:10,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:10,480 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,481 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:10,481 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,481 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a2445dd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,487 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,487 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,487 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,489 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,489 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,494 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:10,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:10,496 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,496 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:10,496 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861370537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861370538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861370543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861370544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,596 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:10,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:10,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:10,600 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:10,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:10,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:10,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:10,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:10,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:10,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:10,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:10,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53eb73e2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,608 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,609 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,613 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304074bae907952c74770827413f6bb7990ec to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074bae907952c74770827413f6bb7990ec 2023-04-07 09:55:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,613 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,613 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,619 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:10,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,624 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:10,625 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:10,625 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,725 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:10,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:10,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:10,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:10,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:10,728 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:10,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:10,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:10,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,730 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,730 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:10,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:10,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:10,731 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,731 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:10,731 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,731 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,734 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c634901[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,738 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,738 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,738 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,740 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,743 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,743 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,743 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,743 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,743 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,746 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,750 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:10,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,750 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:10,750 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:10,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861370844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861370844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,851 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:10,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:10,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861370848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:10,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:10,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861370850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:10,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:10,854 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:10,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:10,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:10,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,856 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,856 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,856 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:10,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:10,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:10,857 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,857 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:10,857 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,858 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555b042f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,869 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,869 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,870 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,872 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,872 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,872 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,872 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,872 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:10,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:10,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:10,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,878 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:10,878 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:10,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,979 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:10,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:10,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:10,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:10,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:10,982 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:10,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:10,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:10,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,983 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:10,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:10,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:10,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:10,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:10,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:10,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:10,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:10,987 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:10,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:10,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:10,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de9a9f4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:10,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:10,992 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,993 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:10,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:10,993 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:10,993 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:10,993 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:10,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:10,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:10,994 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:10,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:10,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:10,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:10,997 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:10,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:10,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:10,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:10,997 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:10,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:10,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:10,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:10,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,004 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,003 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:11,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:11,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,004 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,004 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:11,004 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:11,018 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=57, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/d331b1dc2c184c4f8122bbc41303a051 2023-04-07 09:55:11,104 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:11,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:11,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:11,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:11,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:11,108 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:11,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:11,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:11,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:11,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:11,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:11,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:11,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:11,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:11,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:11,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:11,112 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:11,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:11,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6515a3c8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:11,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:11,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:11,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:11,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:11,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,120 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:11,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:11,123 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:11,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:11,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:11,123 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:11,123 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:11,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,126 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,129 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:11,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:11,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,129 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:11,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,129 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:11,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,230 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:11,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:11,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:11,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:11,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:11,233 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:11,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:11,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:11,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,235 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:11,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:11,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:11,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:11,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:11,238 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:11,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,239 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:11,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23f8c52a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:11,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:11,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,242 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:11,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,242 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:11,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:11,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:11,243 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:11,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:11,246 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:11,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:11,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:11,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:11,246 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:11,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,249 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,249 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,253 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:11,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:11,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,254 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:11,254 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:11,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861371348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:11,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861371351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,354 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:11,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:11,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:11,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:11,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:11,358 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:11,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:11,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:11,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,360 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:11,360 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,360 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:11,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861371357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:11,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861371357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:11,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:11,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:11,362 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:11,362 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,362 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:11,364 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,364 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:11,364 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:11,364 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:11,365 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,365 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:11,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e0fd706[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:11,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:11,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,369 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:11,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:11,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:11,370 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,370 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:11,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:11,373 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:11,373 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:11,373 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:11,373 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:11,373 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:11,374 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,374 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,379 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,379 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:11,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:11,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,379 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:11,379 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:11,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/39a3c0f97dc24a64a2ed404220f3adac 2023-04-07 09:55:11,480 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:11,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:11,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:11,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:11,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:11,483 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:11,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:11,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:11,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,491 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,491 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,491 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:11,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:11,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:11,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:11,495 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:11,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:11,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e097d3a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:11,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:11,498 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:11,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:11,503 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:11,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:11,506 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:11,506 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:11,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:11,506 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:11,506 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:11,506 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:11,507 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,507 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,508 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,511 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,511 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,511 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:11,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,512 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:11,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:11,512 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:11,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,612 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:11,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:11,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:11,616 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:11,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:11,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:11,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,618 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:11,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:11,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:11,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:11,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:11,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:11,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:11,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:11,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:11,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,622 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:11,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ca84d0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:11,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:11,625 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,626 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:11,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:11,626 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:11,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:11,627 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:11,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:11,630 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:11,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:11,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:11,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:11,631 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:11,635 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,635 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,635 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,636 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,637 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,642 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:11,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:11,643 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:11,643 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:11,744 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:11,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:11,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:11,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:11,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:11,753 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:11,753 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,753 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:11,753 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:11,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,755 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,756 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,756 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,756 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:11,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:11,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:11,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:11,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:11,765 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:11,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:11,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:11,766 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:11,768 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:11,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237ec8b6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:11,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:11,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,771 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:11,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:11,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,774 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:11,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:11,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:11,774 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:11,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:11,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:11,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:11,774 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:11,775 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,775 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,776 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:11,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:11,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:11,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:11,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,779 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,779 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:11,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:11,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/c19c309418814d1c87900f5b02bdf22a 2023-04-07 09:55:11,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/d331b1dc2c184c4f8122bbc41303a051 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d331b1dc2c184c4f8122bbc41303a051 2023-04-07 09:55:11,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d331b1dc2c184c4f8122bbc41303a051, entries=200, sequenceid=57, filesize=31.9 K 2023-04-07 09:55:11,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/39a3c0f97dc24a64a2ed404220f3adac as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/39a3c0f97dc24a64a2ed404220f3adac 2023-04-07 09:55:11,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/39a3c0f97dc24a64a2ed404220f3adac, entries=150, sequenceid=57, filesize=11.6 K 2023-04-07 09:55:11,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/c19c309418814d1c87900f5b02bdf22a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/c19c309418814d1c87900f5b02bdf22a 2023-04-07 09:55:11,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/c19c309418814d1c87900f5b02bdf22a, entries=150, sequenceid=57, filesize=11.6 K 2023-04-07 09:55:11,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5aeda58872a183eaf0216fd608afed2c in 1661ms, sequenceid=57, compaction requested=true 2023-04-07 09:55:11,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:11,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:11,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:11,862 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:11,862 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:11,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:11,863 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:11,863 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:11,863 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:11,863 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:11,863 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,863 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,863 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/235ad67f3bc84bf1bd0384927759354d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/b704f6947afe4d9da81dc28959661359, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d331b1dc2c184c4f8122bbc41303a051] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=82.2 K 2023-04-07 09:55:11,863 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/82d0ed8ac6c5466d9e40a9a43c11930d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/12c5a4b9824f4e01ae8b0c9bc39d335b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/39a3c0f97dc24a64a2ed404220f3adac] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=34.8 K 2023-04-07 09:55:11,864 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 235ad67f3bc84bf1bd0384927759354d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680861307927 2023-04-07 09:55:11,864 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 82d0ed8ac6c5466d9e40a9a43c11930d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680861307927 2023-04-07 09:55:11,864 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b704f6947afe4d9da81dc28959661359, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680861307965 2023-04-07 09:55:11,864 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d331b1dc2c184c4f8122bbc41303a051, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680861310086 2023-04-07 09:55:11,865 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 12c5a4b9824f4e01ae8b0c9bc39d335b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680861307965 2023-04-07 09:55:11,866 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 39a3c0f97dc24a64a2ed404220f3adac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680861310086 2023-04-07 09:55:11,874 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:11,879 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:11,880 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#368 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:11,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:11,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:11,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:11,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:11,890 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:11,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:11,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:11,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:11,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:11,895 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:11,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:11,897 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:11,897 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:11,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:11,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:11,900 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:11,900 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:11,900 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:55:11,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:11,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:11,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:11,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:11,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:11,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:12,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:12,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:12,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861372142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861372248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,284 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/da4bbc22e9864acc9905baf920d93cf5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/da4bbc22e9864acc9905baf920d93cf5 2023-04-07 09:55:12,288 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into da4bbc22e9864acc9905baf920d93cf5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:12,289 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:12,289 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=13, startTime=1680861311862; duration=0sec 2023-04-07 09:55:12,289 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:12,289 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:12,291 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:12,291 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:12,291 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:12,292 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1b8afb2f849748bab28858b5d7e4f83f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a8d2bb7d9ecd4d7f89c2455250eef463, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/c19c309418814d1c87900f5b02bdf22a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=34.8 K 2023-04-07 09:55:12,292 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1b8afb2f849748bab28858b5d7e4f83f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680861307927 2023-04-07 09:55:12,292 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a8d2bb7d9ecd4d7f89c2455250eef463, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680861307965 2023-04-07 09:55:12,293 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c19c309418814d1c87900f5b02bdf22a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680861310086 2023-04-07 09:55:12,297 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:12,306 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/a3ac8b523c20408494d414e4811d822b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a3ac8b523c20408494d414e4811d822b 2023-04-07 09:55:12,310 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into a3ac8b523c20408494d414e4811d822b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:12,310 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:12,310 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=13, startTime=1680861311862; duration=0sec 2023-04-07 09:55:12,310 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:12,330 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407762dfa15021243bb915ecd2b8d243eed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407762dfa15021243bb915ecd2b8d243eed 2023-04-07 09:55:12,335 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=77, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/e0f26c1715d345479e0e23ca536a3636 2023-04-07 09:55:12,338 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/9c15264b57dc4e84bf1915a32cc3d959 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/9c15264b57dc4e84bf1915a32cc3d959 2023-04-07 09:55:12,346 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into 9c15264b57dc4e84bf1915a32cc3d959(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:12,346 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:12,346 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=13, startTime=1680861311862; duration=0sec 2023-04-07 09:55:12,346 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:12,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861372356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861372361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861372362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861372369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861372456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,758 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/86068b6bf75247c887557317b093f0ca 2023-04-07 09:55:12,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:12,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861372762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,770 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/8b7ae1c6a41945838f615ff22338bfbd 2023-04-07 09:55:12,776 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/e0f26c1715d345479e0e23ca536a3636 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/e0f26c1715d345479e0e23ca536a3636 2023-04-07 09:55:12,780 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/e0f26c1715d345479e0e23ca536a3636, entries=150, sequenceid=77, filesize=25.1 K 2023-04-07 09:55:12,781 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/86068b6bf75247c887557317b093f0ca as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/86068b6bf75247c887557317b093f0ca 2023-04-07 09:55:12,786 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/86068b6bf75247c887557317b093f0ca, entries=150, sequenceid=77, filesize=11.6 K 2023-04-07 09:55:12,786 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/8b7ae1c6a41945838f615ff22338bfbd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/8b7ae1c6a41945838f615ff22338bfbd 2023-04-07 09:55:12,792 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/8b7ae1c6a41945838f615ff22338bfbd, entries=150, sequenceid=77, filesize=11.6 K 2023-04-07 09:55:12,792 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5aeda58872a183eaf0216fd608afed2c in 892ms, sequenceid=77, compaction requested=false 2023-04-07 09:55:12,792 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:12,792 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:12,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:12,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:12,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:12,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:12,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,800 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:12,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:12,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73999c38[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:12,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:12,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,803 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,803 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:12,803 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:12,803 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:12,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:12,805 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,805 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:12,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:12,807 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:12,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:12,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:12,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:12,808 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:12,812 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,812 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,813 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:12,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:12,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:12,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:12,817 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:12,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:12,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:12,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:12,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,817 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:12,817 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:12,917 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:12,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:12,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:12,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:12,920 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:12,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:12,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:12,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:12,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,921 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:12,922 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,922 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:12,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:12,922 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:12,922 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:12,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:12,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:12,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:12,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:12,924 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:12,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:13,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:13,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:13,341 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040740c26cc143c047ea93dda8cd5ff75859 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040740c26cc143c047ea93dda8cd5ff75859 2023-04-07 09:55:13,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:13,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861373352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:13,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:13,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861373459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:13,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861373661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:13,746 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=92, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/cbf7c985dede49cfbec87d13b62eb1b8 2023-04-07 09:55:13,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:13,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861373965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,156 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/1091efb96fb64d2b9e5cbe70ad2d751b 2023-04-07 09:55:14,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861374363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,368 DEBUG [Thread-2682] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:55:14,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:14,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861374378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,384 DEBUG [Thread-2678] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:55:14,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861374386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861374391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,396 DEBUG [Thread-2676] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:55:14,397 DEBUG [Thread-2674] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:55:14,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:14,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861374475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,567 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/fdeccb4dd55b41b29267010e95edc912 2023-04-07 09:55:14,571 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/cbf7c985dede49cfbec87d13b62eb1b8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/cbf7c985dede49cfbec87d13b62eb1b8 2023-04-07 09:55:14,574 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/cbf7c985dede49cfbec87d13b62eb1b8, entries=150, sequenceid=92, filesize=25.1 K 2023-04-07 09:55:14,575 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/1091efb96fb64d2b9e5cbe70ad2d751b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/1091efb96fb64d2b9e5cbe70ad2d751b 2023-04-07 09:55:14,578 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/1091efb96fb64d2b9e5cbe70ad2d751b, entries=150, sequenceid=92, filesize=11.6 K 2023-04-07 09:55:14,579 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/fdeccb4dd55b41b29267010e95edc912 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fdeccb4dd55b41b29267010e95edc912 2023-04-07 09:55:14,581 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fdeccb4dd55b41b29267010e95edc912, entries=150, sequenceid=92, filesize=11.6 K 2023-04-07 09:55:14,582 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5aeda58872a183eaf0216fd608afed2c in 1658ms, sequenceid=92, compaction requested=true 2023-04-07 09:55:14,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:14,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:14,582 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:14,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:14,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:14,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,584 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:14,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:14,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bcdfdc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:14,585 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:14,586 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,587 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,587 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:14,587 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:14,587 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:14,588 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:14,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:14,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:14,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:14,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:14,589 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:14,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:14,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:14,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:14,589 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:14,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:14,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:14,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:14,593 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:14,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:14,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:14,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:14,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:14,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:14,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:14,595 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:14,596 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:14,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:14,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:14,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:14,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:14,696 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:14,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:14,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:14,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:14,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:14,700 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:14,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:14,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:14,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:14,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:14,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:14,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:14,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:14,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:14,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:14,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:14,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:14,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:14,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:14,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:14,706 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:55:14,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:14,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:14,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:14,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:14,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:14,706 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:14,718 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407b72093faa6cd4e5f931b25ec3d754ffd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b72093faa6cd4e5f931b25ec3d754ffd 2023-04-07 09:55:15,123 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/11e51a84dcb547f1b678d886ddcad082 2023-04-07 09:55:15,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:15,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:15,532 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/3d7fb27a4673454c8051a1b721b3fc8f 2023-04-07 09:55:15,544 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/7917ad1e41bf45d78983781f800eeb74 2023-04-07 09:55:15,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:15,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861375540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,548 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/11e51a84dcb547f1b678d886ddcad082 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/11e51a84dcb547f1b678d886ddcad082 2023-04-07 09:55:15,552 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/11e51a84dcb547f1b678d886ddcad082, entries=150, sequenceid=112, filesize=25.1 K 2023-04-07 09:55:15,553 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/3d7fb27a4673454c8051a1b721b3fc8f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/3d7fb27a4673454c8051a1b721b3fc8f 2023-04-07 09:55:15,558 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/3d7fb27a4673454c8051a1b721b3fc8f, entries=150, sequenceid=112, filesize=11.6 K 2023-04-07 09:55:15,559 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/7917ad1e41bf45d78983781f800eeb74 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/7917ad1e41bf45d78983781f800eeb74 2023-04-07 09:55:15,565 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/7917ad1e41bf45d78983781f800eeb74, entries=150, sequenceid=112, filesize=11.6 K 2023-04-07 09:55:15,566 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5aeda58872a183eaf0216fd608afed2c in 860ms, sequenceid=112, compaction requested=true 2023-04-07 09:55:15,566 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:15,566 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:15,566 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:15,567 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:15,567 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,567 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:15,567 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:15,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:15,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244221df[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:15,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:15,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:15,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,572 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:15,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:15,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:15,573 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,573 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:15,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,573 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,574 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:15,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:15,576 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:15,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:15,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:15,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:15,576 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:15,578 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,579 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,579 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,582 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:15,582 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,582 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:15,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:15,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:15,582 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,583 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:15,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:15,583 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:15,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:15,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:15,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:55:15,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:15,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:15,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:15,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:15,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:15,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:15,683 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:15,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:15,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:15,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:15,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:15,686 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:15,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:15,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:15,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:15,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,688 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:15,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:15,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:15,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:15,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:15,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:15,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:15,690 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:15,690 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:15,690 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:15,690 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:15,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:15,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:15,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:15,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:15,692 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:15,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ac7699[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:15,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:15,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:15,696 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:15,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:15,696 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:15,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:15,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:15,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:15,700 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:15,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:15,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:15,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:15,700 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:15,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:15,702 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:15,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:15,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:15,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:15,713 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:15,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,713 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:15,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:15,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:15,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861375763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,813 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:15,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:15,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:15,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:15,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:15,816 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:15,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:15,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:15,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:15,817 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,817 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:15,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:15,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:15,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:15,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:15,819 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:15,819 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:15,820 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:15,820 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:15,821 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:15,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e6b5cc8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:15,823 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:15,823 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,824 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:15,824 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:15,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:15,827 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:15,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:15,830 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:15,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:15,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:15,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:15,830 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:15,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,833 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:15,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:15,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,836 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,836 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:15,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:15,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:15,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861375871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,937 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:15,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:15,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:15,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:15,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:15,940 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:15,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:15,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:15,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:15,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,941 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:15,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:15,942 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,942 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:15,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:15,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:15,943 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:15,943 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:15,943 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:15,944 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:15,944 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:15,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:15,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:15,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:15,944 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:15,946 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:15,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e7938e6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:15,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:15,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:15,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:15,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:15,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:15,960 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:15,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:15,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:15,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:15,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:15,962 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,962 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:15,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:15,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:15,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:15,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:15,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:15,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:15,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:15,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:15,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:15,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:15,968 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:15,969 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:15,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:15,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:16,065 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407f4529d4a84ee4c3597b9ab8ae26df0ac to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f4529d4a84ee4c3597b9ab8ae26df0ac 2023-04-07 09:55:16,069 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:16,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:16,078 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:16,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:16,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:16,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:16,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,079 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/8e8327c8fffd4a4d9f9b7f85bf81802c 2023-04-07 09:55:16,080 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:16,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:16,080 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,080 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,080 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:16,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:16,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:16,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:16,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:16,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:16,082 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:16,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:16,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861376077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,084 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:16,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:16,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b80be2f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:16,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:16,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,098 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:16,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:16,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:16,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:16,114 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:16,114 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:16,114 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:16,114 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:16,114 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:16,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:16,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:16,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:16,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/42baac8374ac45b8b50fdf9a4c752c28 2023-04-07 09:55:16,127 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:16,127 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:16,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:16,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:16,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:16,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:16,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:16,128 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:16,128 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:16,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:16,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,130 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/e977350673894b0c95f0af3732ad2bda 2023-04-07 09:55:16,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/8e8327c8fffd4a4d9f9b7f85bf81802c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/8e8327c8fffd4a4d9f9b7f85bf81802c 2023-04-07 09:55:16,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/8e8327c8fffd4a4d9f9b7f85bf81802c, entries=200, sequenceid=125, filesize=31.9 K 2023-04-07 09:55:16,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/42baac8374ac45b8b50fdf9a4c752c28 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/42baac8374ac45b8b50fdf9a4c752c28 2023-04-07 09:55:16,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/42baac8374ac45b8b50fdf9a4c752c28, entries=150, sequenceid=125, filesize=11.6 K 2023-04-07 09:55:16,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/e977350673894b0c95f0af3732ad2bda as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e977350673894b0c95f0af3732ad2bda 2023-04-07 09:55:16,228 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:16,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e977350673894b0c95f0af3732ad2bda, entries=150, sequenceid=125, filesize=11.6 K 2023-04-07 09:55:16,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:16,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5aeda58872a183eaf0216fd608afed2c in 580ms, sequenceid=125, compaction requested=true 2023-04-07 09:55:16,231 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:16,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:16,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:16,231 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:55:16,231 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:55:16,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:55:16,233 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59533 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:55:16,233 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 135773 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:55:16,233 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:16,233 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:16,233 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:16,233 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:16,233 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/da4bbc22e9864acc9905baf920d93cf5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/86068b6bf75247c887557317b093f0ca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/1091efb96fb64d2b9e5cbe70ad2d751b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/3d7fb27a4673454c8051a1b721b3fc8f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/42baac8374ac45b8b50fdf9a4c752c28] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=58.1 K 2023-04-07 09:55:16,233 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/9c15264b57dc4e84bf1915a32cc3d959, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/e0f26c1715d345479e0e23ca536a3636, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/cbf7c985dede49cfbec87d13b62eb1b8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/11e51a84dcb547f1b678d886ddcad082, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/8e8327c8fffd4a4d9f9b7f85bf81802c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=132.6 K 2023-04-07 09:55:16,233 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting da4bbc22e9864acc9905baf920d93cf5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680861310086 2023-04-07 09:55:16,233 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 9c15264b57dc4e84bf1915a32cc3d959, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680861310086 2023-04-07 09:55:16,235 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 86068b6bf75247c887557317b093f0ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861310222 2023-04-07 09:55:16,235 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e0f26c1715d345479e0e23ca536a3636, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861310222 2023-04-07 09:55:16,235 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1091efb96fb64d2b9e5cbe70ad2d751b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680861312114 2023-04-07 09:55:16,235 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting cbf7c985dede49cfbec87d13b62eb1b8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680861312114 2023-04-07 09:55:16,236 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3d7fb27a4673454c8051a1b721b3fc8f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680861313338 2023-04-07 09:55:16,236 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 42baac8374ac45b8b50fdf9a4c752c28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861315520 2023-04-07 09:55:16,237 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 11e51a84dcb547f1b678d886ddcad082, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680861313338 2023-04-07 09:55:16,237 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8e8327c8fffd4a4d9f9b7f85bf81802c, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861315520 2023-04-07 09:55:16,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:16,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:16,245 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:16,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:16,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:16,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:16,247 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,247 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:16,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:16,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:16,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:16,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:16,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:16,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:16,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:16,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:16,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:16,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:16,272 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:55:16,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:16,273 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:16,283 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#384 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:16,330 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#383 average throughput is 0.39 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:16,354 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/cdc2c2fe1ced48cbae7a78c4de84edd5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cdc2c2fe1ced48cbae7a78c4de84edd5 2023-04-07 09:55:16,373 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into cdc2c2fe1ced48cbae7a78c4de84edd5(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:16,373 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:16,373 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=11, startTime=1680861316231; duration=0sec 2023-04-07 09:55:16,373 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:16,373 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:55:16,375 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59533 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:55:16,376 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:16,376 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:16,376 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a3ac8b523c20408494d414e4811d822b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/8b7ae1c6a41945838f615ff22338bfbd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fdeccb4dd55b41b29267010e95edc912, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/7917ad1e41bf45d78983781f800eeb74, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e977350673894b0c95f0af3732ad2bda] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=58.1 K 2023-04-07 09:55:16,376 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a3ac8b523c20408494d414e4811d822b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680861310086 2023-04-07 09:55:16,377 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8b7ae1c6a41945838f615ff22338bfbd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861310222 2023-04-07 09:55:16,377 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fdeccb4dd55b41b29267010e95edc912, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680861312114 2023-04-07 09:55:16,377 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7917ad1e41bf45d78983781f800eeb74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680861313338 2023-04-07 09:55:16,378 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e977350673894b0c95f0af3732ad2bda, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861315520 2023-04-07 09:55:16,389 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040758ea43d4c00f4a42bcd94a561f204ca7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040758ea43d4c00f4a42bcd94a561f204ca7 2023-04-07 09:55:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:16,406 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:16,419 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#386 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:16,466 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/b0c0260b53734d0e8d686b1cf1b761b8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/b0c0260b53734d0e8d686b1cf1b761b8 2023-04-07 09:55:16,475 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into b0c0260b53734d0e8d686b1cf1b761b8(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:16,475 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:16,475 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=11, startTime=1680861316231; duration=0sec 2023-04-07 09:55:16,475 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:16,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:16,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861376499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861376607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:16,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45176 deadline: 1680861376813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,861 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/319d0436743942e4bcbf2f15eed276aa 2023-04-07 09:55:16,874 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/8e226cf307f9435791f096a8d33dbeb2 2023-04-07 09:55:16,876 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/d326bd8736f04f149b6cadd4fa8b48bf as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d326bd8736f04f149b6cadd4fa8b48bf 2023-04-07 09:55:16,881 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into d326bd8736f04f149b6cadd4fa8b48bf(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:16,881 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:16,881 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=11, startTime=1680861316231; duration=0sec 2023-04-07 09:55:16,881 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:16,887 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/0460ef743fbd489681afa3ddf555c026 2023-04-07 09:55:16,890 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/319d0436743942e4bcbf2f15eed276aa as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/319d0436743942e4bcbf2f15eed276aa 2023-04-07 09:55:16,896 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/319d0436743942e4bcbf2f15eed276aa, entries=150, sequenceid=144, filesize=25.3 K 2023-04-07 09:55:16,897 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/8e226cf307f9435791f096a8d33dbeb2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8e226cf307f9435791f096a8d33dbeb2 2023-04-07 09:55:16,901 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8e226cf307f9435791f096a8d33dbeb2, entries=150, sequenceid=144, filesize=11.8 K 2023-04-07 09:55:16,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/0460ef743fbd489681afa3ddf555c026 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/0460ef743fbd489681afa3ddf555c026 2023-04-07 09:55:16,905 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/0460ef743fbd489681afa3ddf555c026, entries=150, sequenceid=144, filesize=11.8 K 2023-04-07 09:55:16,915 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5aeda58872a183eaf0216fd608afed2c in 643ms, sequenceid=144, compaction requested=false 2023-04-07 09:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:16,915 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:16,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398a1d78[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:16,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:16,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,924 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,924 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,924 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:16,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,925 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:16,925 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:16,925 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:16,926 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:16,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:16,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:16,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:16,933 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:16,933 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:16,933 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:16,933 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:16,933 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:16,935 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,935 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:16,936 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:16,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,936 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:16,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:16,940 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:16,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:16,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:16,940 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:16,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:16,940 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:16,940 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:16,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:16,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,041 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:17,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:17,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:17,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:17,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:17,054 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:17,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:17,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:17,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,056 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,056 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:17,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:17,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:17,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:17,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:17,070 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:17,070 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,070 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:17,070 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:55:17,070 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:17,071 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:17,084 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407d22b891820aa4710bd8cce497d3a0a13 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d22b891820aa4710bd8cce497d3a0a13 2023-04-07 09:55:17,100 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=160, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/34f94ec2a27d453fb81ac6df427d336b 2023-04-07 09:55:17,137 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/fb304b66fc3b4f51958eb082fdd895c5 2023-04-07 09:55:17,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:17,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:17,178 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/fc8911494277452882edf57d56fefbdf 2023-04-07 09:55:17,183 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/34f94ec2a27d453fb81ac6df427d336b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/34f94ec2a27d453fb81ac6df427d336b 2023-04-07 09:55:17,194 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/34f94ec2a27d453fb81ac6df427d336b, entries=100, sequenceid=160, filesize=18.5 K 2023-04-07 09:55:17,195 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/fb304b66fc3b4f51958eb082fdd895c5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/fb304b66fc3b4f51958eb082fdd895c5 2023-04-07 09:55:17,201 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/fb304b66fc3b4f51958eb082fdd895c5, entries=100, sequenceid=160, filesize=9.4 K 2023-04-07 09:55:17,203 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/fc8911494277452882edf57d56fefbdf as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fc8911494277452882edf57d56fefbdf 2023-04-07 09:55:17,206 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fc8911494277452882edf57d56fefbdf, entries=100, sequenceid=160, filesize=9.4 K 2023-04-07 09:55:17,207 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=13.42 KB/13740 for 5aeda58872a183eaf0216fd608afed2c in 137ms, sequenceid=160, compaction requested=true 2023-04-07 09:55:17,207 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:17,207 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,207 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:17,207 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:17,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:17,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:17,209 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,209 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:17,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d938b9d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:17,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:17,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,214 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:17,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:17,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:17,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:17,216 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:17,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:17,220 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:17,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:17,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:17,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:17,220 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:17,222 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,222 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,222 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,223 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:17,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:17,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,228 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:17,228 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:17,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:17,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:17,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:17,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:17,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:17,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:17,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:17,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:17,301 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407155b8e73ea13410ea19b0598d3462c88 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407155b8e73ea13410ea19b0598d3462c88 2023-04-07 09:55:17,328 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:17,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:17,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:17,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:17,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:17,332 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:17,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:17,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:17,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,336 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:17,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:17,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:17,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:17,336 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:17,336 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:17,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:17,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:17,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:17,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:17,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:17,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:17,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:17,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a7a28c1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:17,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,341 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:17,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:17,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,342 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:17,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:17,346 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:17,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:17,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:17,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:17,346 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:17,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,349 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,349 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:17,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:17,353 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,353 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:17,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,353 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:17,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861317384, max timestamp=1680861317406Num uniques 859; 2023-04-07 09:55:17,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:17,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861317384, max timestamp=1680861317406Num uniques 859; 2023-04-07 09:55:17,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:17,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861317384, max timestamp=1680861317406Num uniques 859; 2023-04-07 09:55:17,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:17,453 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:17,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:17,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:17,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:17,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:17,458 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:17,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:17,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:17,459 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,459 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,460 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:17,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:17,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:17,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:17,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:17,463 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,463 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:17,463 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:17,463 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:17,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:17,468 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:17,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54258067[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:17,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:17,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,474 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:17,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:17,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:17,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:17,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:17,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:17,478 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:17,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:17,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:17,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:17,478 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:17,480 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,480 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,496 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:17,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:17,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,497 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:17,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:17,496 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,497 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,497 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,497 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,497 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:17,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861377539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,597 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:17,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:17,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:17,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:17,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:17,600 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:17,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:17,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:17,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:17,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:17,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:17,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:17,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:17,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:17,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:17,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:17,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:17,603 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,603 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:17,603 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:17,603 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:17,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:17,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d2d2646[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:17,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:17,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,608 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:17,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:17,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:17,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:17,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:17,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:17,611 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:17,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:17,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:17,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:17,611 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:17,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:17,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:17,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,617 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:17,617 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:17,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:17,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861377648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,714 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/25ae5d96cd9f4f3388cab4e19384e4a5 2023-04-07 09:55:17,717 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:17,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:17,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:17,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:17,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:17,721 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:17,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:17,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:17,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,722 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:17,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:17,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:17,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:17,728 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,728 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:17,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/daa5a251b98943f2ba59fcd21beca2fd 2023-04-07 09:55:17,728 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:17,728 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:17,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:17,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,732 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:17,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd39d1d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:17,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:17,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,735 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:17,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:17,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:17,736 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,736 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:17,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:17,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,737 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:17,738 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:17,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:17,741 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:17,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:17,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:17,741 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:17,741 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:17,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/11151e2dbf4445f791df109c8a13757b 2023-04-07 09:55:17,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,743 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/25ae5d96cd9f4f3388cab4e19384e4a5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/25ae5d96cd9f4f3388cab4e19384e4a5 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,748 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,748 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:17,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:17,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:17,749 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:17,749 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:17,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/25ae5d96cd9f4f3388cab4e19384e4a5, entries=250, sequenceid=170, filesize=38.9 K 2023-04-07 09:55:17,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/daa5a251b98943f2ba59fcd21beca2fd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/daa5a251b98943f2ba59fcd21beca2fd 2023-04-07 09:55:17,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/daa5a251b98943f2ba59fcd21beca2fd, entries=150, sequenceid=170, filesize=11.8 K 2023-04-07 09:55:17,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/11151e2dbf4445f791df109c8a13757b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/11151e2dbf4445f791df109c8a13757b 2023-04-07 09:55:17,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/11151e2dbf4445f791df109c8a13757b, entries=150, sequenceid=170, filesize=11.8 K 2023-04-07 09:55:17,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5aeda58872a183eaf0216fd608afed2c in 472ms, sequenceid=170, compaction requested=true 2023-04-07 09:55:17,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:17,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:17,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:17,758 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:17,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:55:17,758 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:17,759 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:17,760 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45874 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:17,760 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:17,760 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:17,760 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,760 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,760 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d326bd8736f04f149b6cadd4fa8b48bf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/319d0436743942e4bcbf2f15eed276aa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/34f94ec2a27d453fb81ac6df427d336b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/25ae5d96cd9f4f3388cab4e19384e4a5] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=108.1 K 2023-04-07 09:55:17,760 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cdc2c2fe1ced48cbae7a78c4de84edd5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8e226cf307f9435791f096a8d33dbeb2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/fb304b66fc3b4f51958eb082fdd895c5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/daa5a251b98943f2ba59fcd21beca2fd] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=44.8 K 2023-04-07 09:55:17,760 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d326bd8736f04f149b6cadd4fa8b48bf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861315520 2023-04-07 09:55:17,760 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cdc2c2fe1ced48cbae7a78c4de84edd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861315520 2023-04-07 09:55:17,760 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 319d0436743942e4bcbf2f15eed276aa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680861315740 2023-04-07 09:55:17,761 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8e226cf307f9435791f096a8d33dbeb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680861315740 2023-04-07 09:55:17,761 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 34f94ec2a27d453fb81ac6df427d336b, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680861316487 2023-04-07 09:55:17,761 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fb304b66fc3b4f51958eb082fdd895c5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680861316487 2023-04-07 09:55:17,761 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 25ae5d96cd9f4f3388cab4e19384e4a5, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680861317148 2023-04-07 09:55:17,761 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting daa5a251b98943f2ba59fcd21beca2fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680861317148 2023-04-07 09:55:17,768 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:17,770 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#395 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:17,849 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:17,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:17,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:17,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:17,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:17,859 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:17,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:17,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:17,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:17,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861317473, max timestamp=1680861317859Num uniques 1018; 2023-04-07 09:55:17,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861317473, max timestamp=1680861317859Num uniques 1018; 2023-04-07 09:55:17,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=102.87 KB heapSize=292.20 KB 2023-04-07 09:55:17,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861317473, max timestamp=1680861317859Num uniques 1018; 2023-04-07 09:55:17,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:55:17,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:17,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:55:17,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:55:17,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:17,863 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:17,863 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:17,863 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:17,863 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:17,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,864 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:17,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:17,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:17,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:17,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:17,865 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,865 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:17,865 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:17,865 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:17,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:17,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:17,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27c49066[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:17,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:17,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,873 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,873 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:17,873 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:17,874 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:17,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:17,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,875 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,875 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,876 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:17,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:17,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:17,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:17,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:17,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:17,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:17,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:17,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:17,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,883 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,890 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304073e50b1b57df6423dac36c99bbe033c3c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073e50b1b57df6423dac36c99bbe033c3c 2023-04-07 09:55:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:17,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:17,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:17,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:17,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:17,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:17,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:17,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:17,893 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:17,893 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:17,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:17,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:17,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:17,902 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=208, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/92e57254ee104860b8ba63bacba811ca 2023-04-07 09:55:17,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861317955, max timestamp=1680861317976Num uniques 859; 2023-04-07 09:55:17,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:17,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861317955, max timestamp=1680861317976Num uniques 859; 2023-04-07 09:55:17,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861317955, max timestamp=1680861317976Num uniques 859; 2023-04-07 09:55:17,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:17,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:17,993 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:17,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:17,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:17,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:17,997 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:17,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:17,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:17,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:17,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:17,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,000 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,007 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,007 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,008 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,008 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,008 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,009 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,009 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,010 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f1a1a41[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,012 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,013 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,013 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,014 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,014 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,016 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,016 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,016 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,016 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,016 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,019 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:18,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,023 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,023 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,023 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,023 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,023 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,023 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,023 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861318050, max timestamp=1680861318111Num uniques 1018; 2023-04-07 09:55:18,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:55:18,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861318050, max timestamp=1680861318111Num uniques 1018; 2023-04-07 09:55:18,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:55:18,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861318050, max timestamp=1680861318111Num uniques 1018; 2023-04-07 09:55:18,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:55:18,123 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:18,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:18,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:18,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:18,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:18,136 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:18,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:18,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:18,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,138 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,138 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,138 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,139 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,146 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,157 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,158 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,158 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,158 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,160 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd118f2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,167 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,171 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,171 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,180 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/28c0b0ae3fce44a6b385c46b50d4e6c1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28c0b0ae3fce44a6b385c46b50d4e6c1 2023-04-07 09:55:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,182 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,182 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,182 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,182 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:18,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,186 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into 28c0b0ae3fce44a6b385c46b50d4e6c1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:18,186 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:18,186 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=12, startTime=1680861317758; duration=0sec 2023-04-07 09:55:18,186 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:18,186 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:18,187 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,187 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,189 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45874 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:18,189 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:18,189 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,189 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/b0c0260b53734d0e8d686b1cf1b761b8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/0460ef743fbd489681afa3ddf555c026, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fc8911494277452882edf57d56fefbdf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/11151e2dbf4445f791df109c8a13757b] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=44.8 K 2023-04-07 09:55:18,190 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b0c0260b53734d0e8d686b1cf1b761b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861315520 2023-04-07 09:55:18,190 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0460ef743fbd489681afa3ddf555c026, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680861315740 2023-04-07 09:55:18,190 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fc8911494277452882edf57d56fefbdf, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680861316487 2023-04-07 09:55:18,190 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 11151e2dbf4445f791df109c8a13757b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680861317148 2023-04-07 09:55:18,201 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#399 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:18,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680861318226, max timestamp=1680861318274Num uniques 1027; 2023-04-07 09:55:18,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 09:55:18,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680861318226, max timestamp=1680861318274Num uniques 1027; 2023-04-07 09:55:18,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680861318226, max timestamp=1680861318274Num uniques 1027; 2023-04-07 09:55:18,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 09:55:18,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-07 09:55:18,287 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:18,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:18,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:18,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:18,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:18,290 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:18,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:18,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:18,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,292 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,292 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,293 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,293 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,294 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,294 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,294 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,294 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,296 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bfa47e0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,298 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,299 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,299 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,300 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,302 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,302 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,307 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,307 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,307 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,312 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:18,312 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,312 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/353f6a021492409faeda6e71570d3e89 2023-04-07 09:55:18,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861318346, max timestamp=1680861318402Num uniques 1036; 2023-04-07 09:55:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861318346, max timestamp=1680861318402Num uniques 1036; 2023-04-07 09:55:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680861318346, max timestamp=1680861318402Num uniques 1036; 2023-04-07 09:55:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-07 09:55:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-07 09:55:18,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-07 09:55:18,412 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:18,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:18,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:18,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:18,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:18,427 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:18,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:18,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:18,427 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,428 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,429 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,432 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,432 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,432 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,432 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,434 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183eb359[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,437 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,438 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,438 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,438 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,438 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,438 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,439 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,442 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,442 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,442 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,442 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,442 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,443 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,444 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,447 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,448 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,448 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861318430, max timestamp=1680861318447Num uniques 1095; 2023-04-07 09:55:18,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-07 09:55:18,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861318430, max timestamp=1680861318447Num uniques 1095; 2023-04-07 09:55:18,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-07 09:55:18,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680861318430, max timestamp=1680861318447Num uniques 1095; 2023-04-07 09:55:18,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-07 09:55:18,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,450 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,450 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,450 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,451 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,451 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:18,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861378473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861378474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680861318461, max timestamp=1680861318483Num uniques 1154; 2023-04-07 09:55:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-07 09:55:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680861318461, max timestamp=1680861318483Num uniques 1154; 2023-04-07 09:55:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-07 09:55:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680861318461, max timestamp=1680861318483Num uniques 1154; 2023-04-07 09:55:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-07 09:55:18,548 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:18,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:18,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:18,551 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:18,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:18,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:18,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,554 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,555 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,556 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,556 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,556 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,556 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,557 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17796a0d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,560 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,564 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,564 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,569 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,572 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,572 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680861318564, max timestamp=1680861318615Num uniques 1113; 2023-04-07 09:55:18,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-07 09:55:18,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680861318564, max timestamp=1680861318615Num uniques 1113; 2023-04-07 09:55:18,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680861318564, max timestamp=1680861318615Num uniques 1113; 2023-04-07 09:55:18,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-07 09:55:18,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-07 09:55:18,620 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/4c4de862b0ad43c1bdd4892715d39d3e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4c4de862b0ad43c1bdd4892715d39d3e 2023-04-07 09:55:18,620 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/511eb425af234404a029bf5bae1387f8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/511eb425af234404a029bf5bae1387f8 2023-04-07 09:55:18,624 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into 511eb425af234404a029bf5bae1387f8(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:18,624 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into 4c4de862b0ad43c1bdd4892715d39d3e(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:18,624 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:18,624 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:18,624 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=12, startTime=1680861317758; duration=0sec 2023-04-07 09:55:18,624 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=12, startTime=1680861317758; duration=0sec 2023-04-07 09:55:18,624 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:18,624 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:18,672 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:18,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:18,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861378670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861378670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680861318652, max timestamp=1680861318672Num uniques 1172; 2023-04-07 09:55:18,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=7872, suffix heap size=149856,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=172 2023-04-07 09:55:18,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680861318652, max timestamp=1680861318672Num uniques 1172; 2023-04-07 09:55:18,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=7872, suffix heap size=149856,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=172 2023-04-07 09:55:18,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680861318652, max timestamp=1680861318672Num uniques 1172; 2023-04-07 09:55:18,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=7872, suffix heap size=149856,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=172 2023-04-07 09:55:18,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:18,686 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:18,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:18,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:18,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,687 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,688 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,690 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,690 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146ce698[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,696 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,696 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,699 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,699 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,700 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,709 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:18,709 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,709 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861318692, max timestamp=1680861318719Num uniques 1031; 2023-04-07 09:55:18,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-07 09:55:18,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861318692, max timestamp=1680861318719Num uniques 1031; 2023-04-07 09:55:18,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-07 09:55:18,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680861318692, max timestamp=1680861318719Num uniques 1031; 2023-04-07 09:55:18,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-07 09:55:18,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/f55d4adec0f146b7b22908fc4387485e 2023-04-07 09:55:18,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/92e57254ee104860b8ba63bacba811ca as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/92e57254ee104860b8ba63bacba811ca 2023-04-07 09:55:18,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/92e57254ee104860b8ba63bacba811ca, entries=150, sequenceid=208, filesize=25.3 K 2023-04-07 09:55:18,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/353f6a021492409faeda6e71570d3e89 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/353f6a021492409faeda6e71570d3e89 2023-04-07 09:55:18,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/353f6a021492409faeda6e71570d3e89, entries=150, sequenceid=208, filesize=11.8 K 2023-04-07 09:55:18,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/f55d4adec0f146b7b22908fc4387485e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f55d4adec0f146b7b22908fc4387485e 2023-04-07 09:55:18,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f55d4adec0f146b7b22908fc4387485e, entries=150, sequenceid=208, filesize=11.8 K 2023-04-07 09:55:18,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=87.22 KB/89310 for 5aeda58872a183eaf0216fd608afed2c in 895ms, sequenceid=208, compaction requested=false 2023-04-07 09:55:18,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:18,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:18,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=107.34 KB heapSize=293.86 KB 2023-04-07 09:55:18,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:18,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:18,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:18,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:18,758 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:18,758 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:18,770 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040768055a4e0e6748289811d18bf1d52329 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040768055a4e0e6748289811d18bf1d52329 2023-04-07 09:55:18,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861378779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861378780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861378781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861378781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861378785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,810 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:18,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:18,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:18,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:18,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:18,815 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:18,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:18,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:18,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,817 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,817 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,824 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,824 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,824 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,824 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de076b2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,844 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,845 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,846 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,848 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,848 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,848 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,848 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,848 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,856 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,858 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:18,858 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861378886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861378887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861378888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861378892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861378892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,958 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:18,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:18,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:18,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:18,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:18,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:18,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:18,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:18,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:18,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:18,964 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:18,965 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,965 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:18,965 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:18,965 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:18,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:18,966 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,966 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:18,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a9b22c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:18,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:18,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:18,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:18,969 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,969 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:18,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:18,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:18,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:18,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:18,975 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:18,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:18,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:18,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:18,976 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:18,977 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,977 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,977 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:18,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,977 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:18,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:18,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:18,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:18,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:18,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:18,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:18,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:18,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:18,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:18,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:18,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:18,982 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:18,982 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:18,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,082 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,085 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,087 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,087 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,087 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,087 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,088 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,088 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,089 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,091 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,091 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdc9f36[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861379091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,096 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861379092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861379093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,099 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,099 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861379098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,102 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861379099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,108 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,108 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,175 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=379, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/1d22056991a041359e492553c3e87239 2023-04-07 09:55:19,208 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,211 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,211 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,211 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,211 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,211 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,211 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,212 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:19,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,213 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,214 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,214 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,214 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,215 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f77ec13[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,217 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,217 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,219 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,219 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,219 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,222 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,222 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,225 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,225 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,225 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,230 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,230 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,330 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,333 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,335 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,336 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,336 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,337 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,339 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41fc4ca9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,342 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,343 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,346 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,346 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,352 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,352 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861379396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861379400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861379401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861379404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861379406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,452 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,456 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,456 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,456 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,456 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,456 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,456 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:19,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,459 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,459 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,459 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,459 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,460 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb9ae68[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,462 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,469 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,470 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,476 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,476 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,576 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,580 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/28df293b92214e4a8cb207b2c179f636 2023-04-07 09:55:19,587 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,589 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,589 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,589 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,589 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3414a3e6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,600 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,600 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,606 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,607 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,607 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,608 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,608 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,707 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,710 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:19,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,714 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,714 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,714 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,714 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,716 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,716 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,716 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1423d603[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,720 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,720 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,720 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,720 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,721 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,721 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,721 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,721 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,723 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,723 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,725 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,725 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,725 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,726 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,730 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,731 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,731 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,831 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,834 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,839 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,839 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,839 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,839 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3719592a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,843 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,845 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,845 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,845 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,845 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,847 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,847 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,854 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,854 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861379904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861379908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861379909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861379910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:19,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861379913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,954 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:19,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:19,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:19,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:19,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:19,956 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:19,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:19,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:19,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:19,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:19,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:19,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:19,959 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:19,960 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:19,960 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:19,960 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:19,960 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:19,961 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46227b4f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:19,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:19,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:19,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:19,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:19,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:19,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:19,972 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:19,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:19,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:19,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:19,972 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:19,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:19,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,975 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:19,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:19,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:19,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:19,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:19,978 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:19,978 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:19,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:19,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:19,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/28c9064b96df4a2eb0ea32bde7e60d9c 2023-04-07 09:55:19,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/1d22056991a041359e492553c3e87239 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1d22056991a041359e492553c3e87239 2023-04-07 09:55:20,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1d22056991a041359e492553c3e87239, entries=150, sequenceid=379, filesize=25.4 K 2023-04-07 09:55:20,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/28df293b92214e4a8cb207b2c179f636 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28df293b92214e4a8cb207b2c179f636 2023-04-07 09:55:20,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28df293b92214e4a8cb207b2c179f636, entries=150, sequenceid=379, filesize=11.9 K 2023-04-07 09:55:20,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/28c9064b96df4a2eb0ea32bde7e60d9c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/28c9064b96df4a2eb0ea32bde7e60d9c 2023-04-07 09:55:20,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/28c9064b96df4a2eb0ea32bde7e60d9c, entries=150, sequenceid=379, filesize=11.9 K 2023-04-07 09:55:20,008 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~293.81 KB/300864, currentSize=73.80 KB/75570 for 5aeda58872a183eaf0216fd608afed2c in 1250ms, sequenceid=379, compaction requested=true 2023-04-07 09:55:20,008 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:20,008 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:20,008 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:20,009 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:20,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:20,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:20,009 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78248 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:20,009 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:20,009 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:20,009 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:20,009 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,009 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,010 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4c4de862b0ad43c1bdd4892715d39d3e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/92e57254ee104860b8ba63bacba811ca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1d22056991a041359e492553c3e87239] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=76.4 K 2023-04-07 09:55:20,010 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28c0b0ae3fce44a6b385c46b50d4e6c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/353f6a021492409faeda6e71570d3e89, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28df293b92214e4a8cb207b2c179f636] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=35.8 K 2023-04-07 09:55:20,010 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4c4de862b0ad43c1bdd4892715d39d3e, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680861317148 2023-04-07 09:55:20,010 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 28c0b0ae3fce44a6b385c46b50d4e6c1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680861317148 2023-04-07 09:55:20,010 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 92e57254ee104860b8ba63bacba811ca, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680861317473 2023-04-07 09:55:20,010 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 353f6a021492409faeda6e71570d3e89, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680861317473 2023-04-07 09:55:20,010 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1d22056991a041359e492553c3e87239, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680861318728 2023-04-07 09:55:20,011 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 28df293b92214e4a8cb207b2c179f636, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680861318728 2023-04-07 09:55:20,017 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:20,019 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#404 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:20,031 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/cca54487be2848f5b2009d81544c84bc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cca54487be2848f5b2009d81544c84bc 2023-04-07 09:55:20,036 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into cca54487be2848f5b2009d81544c84bc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:20,036 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:20,036 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=13, startTime=1680861320008; duration=0sec 2023-04-07 09:55:20,036 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:20,036 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:20,038 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:20,038 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:20,038 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,039 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/511eb425af234404a029bf5bae1387f8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f55d4adec0f146b7b22908fc4387485e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/28c9064b96df4a2eb0ea32bde7e60d9c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=35.8 K 2023-04-07 09:55:20,039 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 511eb425af234404a029bf5bae1387f8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680861317148 2023-04-07 09:55:20,039 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f55d4adec0f146b7b22908fc4387485e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1680861317473 2023-04-07 09:55:20,040 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 28c9064b96df4a2eb0ea32bde7e60d9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680861318728 2023-04-07 09:55:20,044 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/198fb9677a3e49c9ab7aa5af703d4791 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/198fb9677a3e49c9ab7aa5af703d4791 2023-04-07 09:55:20,045 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:20,053 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into 198fb9677a3e49c9ab7aa5af703d4791(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:20,053 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:20,053 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=13, startTime=1680861320008; duration=0sec 2023-04-07 09:55:20,053 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:20,078 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:20,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:20,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:20,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:20,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:20,081 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:20,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:20,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:20,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,083 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:20,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:20,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:20,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:20,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:20,085 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:20,085 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,085 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:20,092 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,092 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:20,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:20,092 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 09:55:20,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:20,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:20,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:20,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:20,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:20,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:20,135 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407f7c47d40ea854ab9a7eaf1d88bf5233f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f7c47d40ea854ab9a7eaf1d88bf5233f 2023-04-07 09:55:20,154 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/4f733dfcb0f741ec9c909bdba32a5a0a 2023-04-07 09:55:20,174 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/8af65f739c7e4d3db4edde9fbca80f7e 2023-04-07 09:55:20,459 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/e6b1fd30c87545218ea2160d432a3dbe as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e6b1fd30c87545218ea2160d432a3dbe 2023-04-07 09:55:20,462 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into e6b1fd30c87545218ea2160d432a3dbe(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:20,463 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:20,463 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=13, startTime=1680861320009; duration=0sec 2023-04-07 09:55:20,463 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:20,590 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/468f2cbbc80e4438942b5fd6e16f3e32 2023-04-07 09:55:20,594 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/4f733dfcb0f741ec9c909bdba32a5a0a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4f733dfcb0f741ec9c909bdba32a5a0a 2023-04-07 09:55:20,598 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4f733dfcb0f741ec9c909bdba32a5a0a, entries=150, sequenceid=395, filesize=25.4 K 2023-04-07 09:55:20,598 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/8af65f739c7e4d3db4edde9fbca80f7e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8af65f739c7e4d3db4edde9fbca80f7e 2023-04-07 09:55:20,601 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8af65f739c7e4d3db4edde9fbca80f7e, entries=150, sequenceid=395, filesize=11.9 K 2023-04-07 09:55:20,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/468f2cbbc80e4438942b5fd6e16f3e32 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/468f2cbbc80e4438942b5fd6e16f3e32 2023-04-07 09:55:20,605 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/468f2cbbc80e4438942b5fd6e16f3e32, entries=150, sequenceid=395, filesize=11.9 K 2023-04-07 09:55:20,606 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 5aeda58872a183eaf0216fd608afed2c in 514ms, sequenceid=395, compaction requested=false 2023-04-07 09:55:20,606 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:20,606 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:20,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:20,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:20,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:20,608 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5738fbfe[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:20,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:20,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:20,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:20,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:20,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:20,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:20,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:20,615 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:20,615 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:20,618 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,618 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,618 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,619 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:20,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:20,622 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:20,622 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:20,622 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:20,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:20,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:20,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:20,722 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:20,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:20,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:20,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:20,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:20,727 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:20,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:20,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:20,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:20,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:20,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:20,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,731 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:20,740 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,740 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:20,740 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:20,740 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:20,742 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,742 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d21bcd4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:20,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:20,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,747 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:20,748 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,748 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:20,749 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:20,752 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:20,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:20,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:20,752 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:20,752 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:20,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:20,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,755 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:20,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:20,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:20,757 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:20,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,758 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,758 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,758 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:20,757 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:20,758 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:20,758 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:20,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:20,858 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:20,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:20,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:20,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:20,862 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,864 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,864 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:20,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:20,864 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,864 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:20,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:20,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:20,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:20,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:20,866 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,866 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:20,866 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:20,866 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:20,868 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,868 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:20,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200c6e53[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:20,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:20,870 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,871 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:20,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,871 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:20,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:20,871 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:20,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,878 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,878 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,878 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:20,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:20,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:20,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:20,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:20,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:20,880 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:20,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:20,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,882 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,882 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,884 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,886 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:20,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:20,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:20,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:20,886 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,886 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:20,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:20,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:20,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:20,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:20,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:20,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:20,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:20,986 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:20,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:20,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:20,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:20,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:20,989 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:20,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:20,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:20,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,993 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:20,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:20,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:20,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:20,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:20,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:20,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:20,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:20,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:20,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:20,994 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:20,994 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,994 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:20,995 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:20,995 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:20,996 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:20,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:20,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:20,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7521811e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:20,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,001 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,005 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,005 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,006 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,008 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,008 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,008 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,009 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861320995, max timestamp=1680861321009Num uniques 859; 2023-04-07 09:55:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861320995, max timestamp=1680861321009Num uniques 859; 2023-04-07 09:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861320995, max timestamp=1680861321009Num uniques 859; 2023-04-07 09:55:21,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:21,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,014 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,014 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,014 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:21,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,014 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,015 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861381029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861381029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861381030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861381031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861381032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,114 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:21,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:21,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:21,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:21,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:21,118 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:21,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:21,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:21,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,120 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,120 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,120 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:21,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:21,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:21,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:21,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:21,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:21,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:21,123 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,123 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,123 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,123 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:21,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d6634f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:21,125 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,125 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,127 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,128 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,130 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,130 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,132 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,133 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,133 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,137 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,137 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,137 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:21,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861381140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861381141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861381142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861381142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861381142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,238 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:21,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:21,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:21,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:21,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:21,241 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:21,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:21,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:21,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:21,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:21,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:21,250 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,250 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:21,250 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:21,250 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,252 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:21,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3541b43d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:21,253 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,258 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,258 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,258 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,258 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,258 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,261 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,261 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,261 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,261 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,262 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,273 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,273 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,274 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861381342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861381348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861381349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861381349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861381350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,374 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:21,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:21,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:21,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:21,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:21,377 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:21,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:21,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:21,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,377 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,378 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,378 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,379 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,379 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:21,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:21,380 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,380 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:21,380 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:21,380 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:21,382 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,382 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,382 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,382 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:21,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76fc3a30[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:21,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,385 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,385 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,387 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,390 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a842fb5d5bdc45e89c3f8b9660ee2045 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a842fb5d5bdc45e89c3f8b9660ee2045 2023-04-07 09:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,390 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,390 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,390 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,390 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,390 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,392 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,392 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,392 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,403 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,403 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,403 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,404 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,503 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:21,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:21,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:21,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:21,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:21,507 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:21,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:21,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:21,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,508 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,509 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:21,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:21,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:21,510 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,510 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:21,510 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:21,510 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,513 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fac7c80[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:21,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,515 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,516 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,516 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,520 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,520 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,521 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,523 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,526 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,526 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,526 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,526 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,526 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,526 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,526 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,526 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,526 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,527 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,527 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,527 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,627 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:21,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:21,630 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:21,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:21,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:21,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,630 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,632 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,632 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:21,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:21,633 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,633 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:21,633 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:21,634 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:21,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:21,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:21,634 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:21,635 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:21,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59b6ba21[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:21,637 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,637 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,638 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,638 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,638 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,647 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,647 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,647 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,647 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,647 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,649 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,649 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,649 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,650 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861381646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,655 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,655 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:21,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861381653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861381653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861381654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:21,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861381655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,755 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:21,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:21,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:21,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:21,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:21,768 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:21,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:21,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:21,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:21,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:21,776 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:21,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@616e0d76[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:21,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,780 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,782 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,784 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,784 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,788 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,788 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,788 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,792 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,792 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,795 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=408, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/80ac3f86811d4c3ab499d87c0c52dc61 2023-04-07 09:55:21,892 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:21,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:21,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:21,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:21,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:21,896 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:21,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:21,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:21,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,897 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,897 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,898 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:21,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:21,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:21,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:21,899 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,899 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:21,899 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:21,899 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,901 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:21,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@765496d1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:21,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:21,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:21,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,907 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:21,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:21,911 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:21,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:21,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:21,911 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:21,911 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:21,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:21,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:21,913 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:21,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:21,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:21,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:21,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:21,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:21,916 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:21,916 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:21,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:22,017 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:22,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:22,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:22,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:22,020 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:22,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:22,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:22,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:22,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,023 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:22,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:22,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:22,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:22,025 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,025 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:22,025 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:22,025 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:22,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:22,026 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:22,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:22,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:22,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:22,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af0ace7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:22,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:22,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:22,030 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:22,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:22,033 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:22,033 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:22,033 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:22,033 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:22,033 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:22,035 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,035 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,035 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:22,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:22,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:22,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:22,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:22,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:22,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:22,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:22,040 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:22,040 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:22,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,141 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:22,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:22,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:22,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:22,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:22,144 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:22,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:22,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:22,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:22,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:22,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:22,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:22,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:22,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:22,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:22,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:22,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:22,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:22,154 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:22,157 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:22,157 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:22,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@767b6a2a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:22,159 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:22,159 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,160 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:22,161 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:22,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:22,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:22,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861382157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:22,164 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:22,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861382162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:22,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:22,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861382162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:22,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861382162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:22,167 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:22,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:22,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:22,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:22,167 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:22,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:22,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:22,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:22,169 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:22,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:22,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861382164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:22,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:22,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:22,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:22,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:22,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:22,179 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,180 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:22,180 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:22,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:22,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:22,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/101828e0845c4749a1d52fd3a9d6d7bc 2023-04-07 09:55:22,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/e9f7f79c0261426a8719e08f22777551 2023-04-07 09:55:22,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/80ac3f86811d4c3ab499d87c0c52dc61 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/80ac3f86811d4c3ab499d87c0c52dc61 2023-04-07 09:55:22,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/80ac3f86811d4c3ab499d87c0c52dc61, entries=350, sequenceid=408, filesize=52.9 K 2023-04-07 09:55:22,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/101828e0845c4749a1d52fd3a9d6d7bc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/101828e0845c4749a1d52fd3a9d6d7bc 2023-04-07 09:55:22,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/101828e0845c4749a1d52fd3a9d6d7bc, entries=150, sequenceid=408, filesize=11.9 K 2023-04-07 09:55:22,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/e9f7f79c0261426a8719e08f22777551 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e9f7f79c0261426a8719e08f22777551 2023-04-07 09:55:22,250 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e9f7f79c0261426a8719e08f22777551, entries=150, sequenceid=408, filesize=11.9 K 2023-04-07 09:55:22,251 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 5aeda58872a183eaf0216fd608afed2c in 1279ms, sequenceid=408, compaction requested=true 2023-04-07 09:55:22,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:22,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:22,251 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:22,251 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:22,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:22,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:22,255 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:22,255 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:22,255 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:22,255 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:22,255 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,256 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,256 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cca54487be2848f5b2009d81544c84bc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8af65f739c7e4d3db4edde9fbca80f7e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/101828e0845c4749a1d52fd3a9d6d7bc] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.2 K 2023-04-07 09:55:22,256 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/198fb9677a3e49c9ab7aa5af703d4791, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4f733dfcb0f741ec9c909bdba32a5a0a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/80ac3f86811d4c3ab499d87c0c52dc61] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=104.3 K 2023-04-07 09:55:22,256 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cca54487be2848f5b2009d81544c84bc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680861318728 2023-04-07 09:55:22,256 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 198fb9677a3e49c9ab7aa5af703d4791, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680861318728 2023-04-07 09:55:22,257 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4f733dfcb0f741ec9c909bdba32a5a0a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680861318776 2023-04-07 09:55:22,257 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 80ac3f86811d4c3ab499d87c0c52dc61, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680861320956 2023-04-07 09:55:22,258 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8af65f739c7e4d3db4edde9fbca80f7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680861318776 2023-04-07 09:55:22,258 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 101828e0845c4749a1d52fd3a9d6d7bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680861320959 2023-04-07 09:55:22,266 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#414 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:22,268 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#413 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:22,280 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:22,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:22,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:22,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:22,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:22,284 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:22,284 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:22,284 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:22,284 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:22,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:22,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:22,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:22,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:22,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:22,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:22,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:22,287 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:22,287 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:22,288 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-07 09:55:22,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:22,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:22,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:22,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:22,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:22,288 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:22,319 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304075ac58e959734406bac0c7ffc11c666c7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075ac58e959734406bac0c7ffc11c666c7 2023-04-07 09:55:22,325 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=443, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/4abfef2f8fd041f88c7be578e9cbe321 2023-04-07 09:55:22,683 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/6030cd46ae074333a4da8ca9e9012cba as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/6030cd46ae074333a4da8ca9e9012cba 2023-04-07 09:55:22,687 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into 6030cd46ae074333a4da8ca9e9012cba(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:22,687 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:22,687 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=13, startTime=1680861322251; duration=0sec 2023-04-07 09:55:22,688 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:22,688 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:22,689 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:22,689 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:22,689 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:22,689 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e6b1fd30c87545218ea2160d432a3dbe, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/468f2cbbc80e4438942b5fd6e16f3e32, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e9f7f79c0261426a8719e08f22777551] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.2 K 2023-04-07 09:55:22,690 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e6b1fd30c87545218ea2160d432a3dbe, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680861318728 2023-04-07 09:55:22,690 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 468f2cbbc80e4438942b5fd6e16f3e32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680861318776 2023-04-07 09:55:22,690 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e9f7f79c0261426a8719e08f22777551, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680861320959 2023-04-07 09:55:22,695 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:22,716 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/0f6abe398b9b485fb9f4473c0057d9ec as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/0f6abe398b9b485fb9f4473c0057d9ec 2023-04-07 09:55:22,720 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into 0f6abe398b9b485fb9f4473c0057d9ec(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:22,720 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:22,720 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=13, startTime=1680861322251; duration=0sec 2023-04-07 09:55:22,720 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:22,738 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/cd99444003714124adb37c5943f0e32b 2023-04-07 09:55:23,105 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/578317b0997c407ba680bbb6495bc7fa as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/578317b0997c407ba680bbb6495bc7fa 2023-04-07 09:55:23,109 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into 578317b0997c407ba680bbb6495bc7fa(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:23,109 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:23,109 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=13, startTime=1680861322251; duration=0sec 2023-04-07 09:55:23,109 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:23,148 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/30a25c7002294794a86627feda000f5f 2023-04-07 09:55:23,151 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/4abfef2f8fd041f88c7be578e9cbe321 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4abfef2f8fd041f88c7be578e9cbe321 2023-04-07 09:55:23,155 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4abfef2f8fd041f88c7be578e9cbe321, entries=150, sequenceid=443, filesize=25.4 K 2023-04-07 09:55:23,156 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/cd99444003714124adb37c5943f0e32b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cd99444003714124adb37c5943f0e32b 2023-04-07 09:55:23,160 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cd99444003714124adb37c5943f0e32b, entries=150, sequenceid=443, filesize=11.9 K 2023-04-07 09:55:23,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/30a25c7002294794a86627feda000f5f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/30a25c7002294794a86627feda000f5f 2023-04-07 09:55:23,164 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/30a25c7002294794a86627feda000f5f, entries=150, sequenceid=443, filesize=11.9 K 2023-04-07 09:55:23,165 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=0 B/0 for 5aeda58872a183eaf0216fd608afed2c in 877ms, sequenceid=443, compaction requested=false 2023-04-07 09:55:23,165 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:23,165 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:23,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:23,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@676fdc2e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:23,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:23,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:23,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:23,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:23,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:23,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:23,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:23,186 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:23,186 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:23,186 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:23,186 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:23,186 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:23,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,188 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:23,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:23,193 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,193 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:23,193 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:23,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:23,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:23,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:23,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:23,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:23,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:23,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:23,240 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e6bab95b886b4a06b95b77ab9ded71e3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e6bab95b886b4a06b95b77ab9ded71e3 2023-04-07 09:55:23,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861323245, max timestamp=1680861323280Num uniques 859; 2023-04-07 09:55:23,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:23,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861323245, max timestamp=1680861323280Num uniques 859; 2023-04-07 09:55:23,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:23,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861323245, max timestamp=1680861323280Num uniques 859; 2023-04-07 09:55:23,282 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:23,293 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:23,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:23,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:23,296 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:23,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:23,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:23,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,297 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:23,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:23,299 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,299 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:23,299 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:23,299 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:23,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:23,301 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,301 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,302 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,302 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:23,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@355f2c0c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:23,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:23,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,305 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:23,305 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:23,305 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:23,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:23,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:23,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:23,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:23,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861383304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:23,310 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:23,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:23,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:23,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:23,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,311 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:23,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861383304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,313 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861383306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861383306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861383306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,317 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,317 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,317 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:23,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:23,317 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:23,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,318 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,318 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,318 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,318 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:23,318 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,318 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:23,318 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861383412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861383412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,418 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:23,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:23,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:23,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861383415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861383415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861383416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:23,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:23,421 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:23,421 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,421 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:23,421 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:23,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,423 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,423 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,423 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,423 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:23,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:23,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,425 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:23,425 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:23,425 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:23,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:23,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad739d4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:23,428 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:23,428 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:23,433 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,433 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:23,434 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:23,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:23,436 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:23,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:23,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:23,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:23,436 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:23,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,438 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,440 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,442 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:23,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,442 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:23,442 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:23,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,542 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:23,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:23,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:23,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:23,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:23,545 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:23,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:23,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:23,546 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,546 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,548 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:23,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:23,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:23,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:23,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:23,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:23,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:23,550 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,552 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:23,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7503c9e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:23,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:23,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:23,555 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,555 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:23,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:23,559 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:23,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:23,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:23,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:23,559 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:23,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,564 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:23,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:23,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:23,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,566 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:23,566 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:23,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861383618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861383619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861383622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861383622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861383623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,650 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=456, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/13763ac236de46fe8fdefefebc42de4a 2023-04-07 09:55:23,667 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:23,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:23,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:23,670 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:23,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:23,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:23,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:23,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:23,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:23,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:23,675 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,675 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:23,675 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:23,675 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:23,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:23,678 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,678 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,678 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,678 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:23,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ec71ef6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:23,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:23,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,682 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:23,682 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:23,682 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,683 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:23,683 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,684 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:23,684 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:23,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:23,693 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:23,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:23,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:23,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:23,694 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:23,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,695 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,696 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:23,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:23,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,701 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:23,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,702 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:23,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,702 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,702 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,702 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,702 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,802 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:23,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:23,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:23,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:23,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:23,807 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:23,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:23,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:23,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,810 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,810 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,810 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,810 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:23,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:23,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:23,814 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:23,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b46e6e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:23,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:23,816 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,817 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:23,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:23,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:23,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:23,823 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:23,826 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:23,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:23,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:23,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:23,826 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:23,831 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:23,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:23,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,840 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:23,840 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:23,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861383925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861383926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861383930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861383931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861383931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,940 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:23,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:23,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:23,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:23,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:23,944 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:23,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:23,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:23,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,946 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,946 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:23,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:23,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:23,952 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,952 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:23,952 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:23,952 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:23,954 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:23,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bee475[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:23,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:23,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:23,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:23,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:23,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:23,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:23,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:23,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:23,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:23,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:23,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:23,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:23,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,970 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,970 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:23,971 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:23,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:23,970 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:23,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:23,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:23,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:23,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:23,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:23,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:23,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:23,971 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:23,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:23,971 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:24,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/4e249a939e2a416b8e557155fd09e077 2023-04-07 09:55:24,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/56d9551739aa4dd097611dd666fe96a6 2023-04-07 09:55:24,071 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:24,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:24,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:24,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/13763ac236de46fe8fdefefebc42de4a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/13763ac236de46fe8fdefefebc42de4a 2023-04-07 09:55:24,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:24,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:24,075 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:24,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:24,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:24,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:24,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,075 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,076 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,076 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:24,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:24,077 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,077 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:24,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:24,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/13763ac236de46fe8fdefefebc42de4a, entries=350, sequenceid=456, filesize=52.9 K 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:24,079 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:24,079 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:24,079 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:24,079 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:24,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:24,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/4e249a939e2a416b8e557155fd09e077 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/4e249a939e2a416b8e557155fd09e077 2023-04-07 09:55:24,081 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,081 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@478c9410[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:24,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/4e249a939e2a416b8e557155fd09e077, entries=150, sequenceid=456, filesize=11.9 K 2023-04-07 09:55:24,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:24,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/56d9551739aa4dd097611dd666fe96a6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/56d9551739aa4dd097611dd666fe96a6 2023-04-07 09:55:24,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,086 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:24,086 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:24,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:24,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:24,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/56d9551739aa4dd097611dd666fe96a6, entries=150, sequenceid=456, filesize=11.9 K 2023-04-07 09:55:24,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:24,087 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:24,088 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:24,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 5aeda58872a183eaf0216fd608afed2c in 866ms, sequenceid=456, compaction requested=true 2023-04-07 09:55:24,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:24,088 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:24,088 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:24,088 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:24,088 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:24,088 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:24,090 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,090 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:24,090 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:24,090 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:24,090 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:24,090 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/0f6abe398b9b485fb9f4473c0057d9ec, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4abfef2f8fd041f88c7be578e9cbe321, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/13763ac236de46fe8fdefefebc42de4a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=104.4 K 2023-04-07 09:55:24,090 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:24,090 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/6030cd46ae074333a4da8ca9e9012cba, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cd99444003714124adb37c5943f0e32b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/4e249a939e2a416b8e557155fd09e077] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.3 K 2023-04-07 09:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:24,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:24,090 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:24,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:24,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:24,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:24,090 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 0f6abe398b9b485fb9f4473c0057d9ec, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680861320959 2023-04-07 09:55:24,091 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:24,091 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 6030cd46ae074333a4da8ca9e9012cba, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680861320959 2023-04-07 09:55:24,091 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4abfef2f8fd041f88c7be578e9cbe321, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680861321020 2023-04-07 09:55:24,091 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cd99444003714124adb37c5943f0e32b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680861321020 2023-04-07 09:55:24,091 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 13763ac236de46fe8fdefefebc42de4a, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680861323206 2023-04-07 09:55:24,091 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4e249a939e2a416b8e557155fd09e077, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680861323209 2023-04-07 09:55:24,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:24,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:24,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:24,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:24,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:24,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,099 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:24,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:24,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:24,099 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:24,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:24,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:24,100 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:24,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:24,100 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:24,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:24,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:24,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,105 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:24,111 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#422 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:24,200 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:24,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:24,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:24,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:24,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:24,203 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:24,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:24,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:24,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:24,204 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,204 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:24,206 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:24,206 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,206 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:24,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:24,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:24,208 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:24,208 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:24,208 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-07 09:55:24,208 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:24,209 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:24,209 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:24,209 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:24,209 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:24,209 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:24,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:24,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:24,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861384443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861384450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861384450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861384451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861384452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,518 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/b30879dfdd854e1d99865a2b2d49b8af as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b30879dfdd854e1d99865a2b2d49b8af 2023-04-07 09:55:24,521 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into b30879dfdd854e1d99865a2b2d49b8af(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:24,522 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:24,522 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=13, startTime=1680861324088; duration=0sec 2023-04-07 09:55:24,522 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:24,522 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:24,523 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:24,523 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:24,523 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:24,523 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/578317b0997c407ba680bbb6495bc7fa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/30a25c7002294794a86627feda000f5f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/56d9551739aa4dd097611dd666fe96a6] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.3 K 2023-04-07 09:55:24,523 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 578317b0997c407ba680bbb6495bc7fa, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680861320959 2023-04-07 09:55:24,523 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 30a25c7002294794a86627feda000f5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680861321020 2023-04-07 09:55:24,524 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 56d9551739aa4dd097611dd666fe96a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680861323209 2023-04-07 09:55:24,532 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#425 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:24,551 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/1f5a1cd7ba354214a30fdb9b41c7925c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1f5a1cd7ba354214a30fdb9b41c7925c 2023-04-07 09:55:24,553 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/dfdbaa976e8643b0aebf7d4008ebe4fd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dfdbaa976e8643b0aebf7d4008ebe4fd 2023-04-07 09:55:24,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861384555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,557 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into 1f5a1cd7ba354214a30fdb9b41c7925c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:24,558 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:24,558 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=13, startTime=1680861324088; duration=0sec 2023-04-07 09:55:24,558 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:24,559 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into dfdbaa976e8643b0aebf7d4008ebe4fd(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:24,559 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:24,559 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=13, startTime=1680861324088; duration=0sec 2023-04-07 09:55:24,559 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:24,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861384556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861384559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861384559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861384559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,621 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304073c5c2f147111421ba2fcdb88b5e62028 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073c5c2f147111421ba2fcdb88b5e62028 2023-04-07 09:55:24,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861384757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861384762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861384763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861384764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:24,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861384764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,026 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=491, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/dc7f0f8d140542b283fd47dbcce8a241 2023-04-07 09:55:25,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861385063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861385067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45164 deadline: 1680861385067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861385067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45118 deadline: 1680861385068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,436 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/069114fb05144a0386c42d9079de4346 2023-04-07 09:55:25,445 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/20eaf73efe2743d7be546e3d07f33ceb 2023-04-07 09:55:25,448 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/dc7f0f8d140542b283fd47dbcce8a241 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dc7f0f8d140542b283fd47dbcce8a241 2023-04-07 09:55:25,452 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dc7f0f8d140542b283fd47dbcce8a241, entries=150, sequenceid=491, filesize=25.4 K 2023-04-07 09:55:25,452 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/069114fb05144a0386c42d9079de4346 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/069114fb05144a0386c42d9079de4346 2023-04-07 09:55:25,455 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/069114fb05144a0386c42d9079de4346, entries=150, sequenceid=491, filesize=11.9 K 2023-04-07 09:55:25,456 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/20eaf73efe2743d7be546e3d07f33ceb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20eaf73efe2743d7be546e3d07f33ceb 2023-04-07 09:55:25,459 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20eaf73efe2743d7be546e3d07f33ceb, entries=150, sequenceid=491, filesize=11.9 K 2023-04-07 09:55:25,460 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 5aeda58872a183eaf0216fd608afed2c in 1251ms, sequenceid=491, compaction requested=false 2023-04-07 09:55:25,460 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:25,460 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619a12b9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:25,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:25,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:25,465 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:25,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:25,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:25,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:25,468 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:25,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:25,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:25,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:25,469 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:25,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,473 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:25,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:25,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,476 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:25,476 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:25,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:25,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:25,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:55:25,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:25,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:25,576 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:25,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:25,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:25,579 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:25,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:25,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:25,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,580 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,581 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,581 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:25,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:25,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:25,583 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,583 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:25,583 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:25,583 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:25,584 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407402a5dd03b344bca8de1d265746076df to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407402a5dd03b344bca8de1d265746076df 2023-04-07 09:55:25,585 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,585 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10acd7fa[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:25,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:25,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,589 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:25,590 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:25,590 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,590 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:25,592 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,592 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:25,594 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:25,594 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:25,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:25,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:25,595 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:25,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,596 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680861325589, max timestamp=1680861325598Num uniques 859; 2023-04-07 09:55:25,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-07 09:55:25,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680861325589, max timestamp=1680861325598Num uniques 859; 2023-04-07 09:55:25,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680861325589, max timestamp=1680861325598Num uniques 859; 2023-04-07 09:55:25,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-07 09:55:25,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-07 09:55:25,601 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:25,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:25,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,603 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:25,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,603 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:25,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861385618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861385619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1680861385619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861385620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861385622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,703 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:25,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:25,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:25,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:25,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:25,706 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:25,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:25,706 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:25,707 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,707 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:25,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:25,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:25,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:25,709 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:25,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:25,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:25,710 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:25,712 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:25,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:25,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@268704d6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:25,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,715 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:25,715 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,715 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,716 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:25,720 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:25,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:25,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:25,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:25,720 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:25,721 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,721 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,721 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,722 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861385725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:25,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:25,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:25,730 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:25,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,730 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:25,730 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861385728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861385728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861385728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1680861385729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,830 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:25,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:25,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:25,834 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:25,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:25,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:25,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,835 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:25,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:25,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:25,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:25,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5082802[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:25,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:25,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:25,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:25,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:25,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:25,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:25,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,846 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:25,851 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:25,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:25,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:25,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:25,851 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:25,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:25,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:25,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:25,863 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:25,863 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:25,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861385928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861385933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861385934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861385934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1680861385936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,963 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:25,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:25,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:25,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:25,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:25,966 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:25,966 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,966 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:25,966 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:25,966 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,966 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,969 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,969 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:25,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:25,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:25,970 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,970 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:25,970 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:25,970 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:25,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:25,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:25,972 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:25,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3436fdc0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:25,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:25,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,976 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:25,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:25,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:25,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:25,977 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:25,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:25,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:25,979 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:25,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:25,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:25,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:25,979 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:25,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:25,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:25,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:25,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:25,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:25,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:25,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:25,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:25,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,986 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:25,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:25,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:25,987 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:25,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:25,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:25,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:25,989 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=505, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/50a7468b0cb04c8aba3b59412d0a6ab5 2023-04-07 09:55:26,087 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:26,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:26,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:26,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:26,090 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:26,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:26,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:26,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,091 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:26,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:26,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:26,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:26,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:26,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:26,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:26,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:26,095 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:26,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4253efbb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:26,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:26,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,100 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:26,100 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:26,100 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:26,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,100 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:26,101 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:26,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:26,103 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:26,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:26,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:26,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:26,104 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:26,105 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,105 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,105 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:26,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:26,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:26,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:26,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:26,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:26,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:26,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:26,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,109 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:26,109 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:26,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:26,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:26,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,209 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:26,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:26,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:26,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:26,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:26,212 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:26,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:26,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:26,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,212 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,214 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:26,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:26,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:26,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:26,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:26,216 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,216 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:26,216 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:26,216 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:26,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:26,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,218 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c6cf293[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:26,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:26,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:26,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:26,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:26,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:26,223 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:26,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:26,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:26,225 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:26,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:26,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:26,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:26,225 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:26,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:26,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:26,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:26,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:26,231 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,231 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861386236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861386240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861386241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861386244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1680861386245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,331 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:26,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:26,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:26,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:26,334 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:26,334 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,334 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:26,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:26,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:26,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:26,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:26,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:26,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:26,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:26,342 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,342 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18609924[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:26,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:26,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,345 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:26,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:26,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:26,347 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:26,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:26,350 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:26,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:26,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:26,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:26,350 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:26,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:26,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:26,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:26,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,357 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:26,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:26,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:26,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:26,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:26,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:26,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:26,362 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:26,362 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:26,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:26,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:26,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/b3b653250ff94435a4eb7730d2202290 2023-04-07 09:55:26,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/849bb459baa34609894f448ce7e5c99a 2023-04-07 09:55:26,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/50a7468b0cb04c8aba3b59412d0a6ab5 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/50a7468b0cb04c8aba3b59412d0a6ab5 2023-04-07 09:55:26,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/50a7468b0cb04c8aba3b59412d0a6ab5, entries=200, sequenceid=505, filesize=32.3 K 2023-04-07 09:55:26,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/b3b653250ff94435a4eb7730d2202290 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b3b653250ff94435a4eb7730d2202290 2023-04-07 09:55:26,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b3b653250ff94435a4eb7730d2202290, entries=150, sequenceid=505, filesize=11.9 K 2023-04-07 09:55:26,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/849bb459baa34609894f448ce7e5c99a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/849bb459baa34609894f448ce7e5c99a 2023-04-07 09:55:26,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/849bb459baa34609894f448ce7e5c99a, entries=150, sequenceid=505, filesize=11.9 K 2023-04-07 09:55:26,425 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 5aeda58872a183eaf0216fd608afed2c in 853ms, sequenceid=505, compaction requested=true 2023-04-07 09:55:26,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:26,426 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:26,425 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:26,426 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:26,426 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:26,426 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85894 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:26,426 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:26,427 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:26,427 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,427 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,427 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b30879dfdd854e1d99865a2b2d49b8af, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/069114fb05144a0386c42d9079de4346, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b3b653250ff94435a4eb7730d2202290] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.4 K 2023-04-07 09:55:26,427 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dfdbaa976e8643b0aebf7d4008ebe4fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dc7f0f8d140542b283fd47dbcce8a241, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/50a7468b0cb04c8aba3b59412d0a6ab5] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=83.9 K 2023-04-07 09:55:26,427 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b30879dfdd854e1d99865a2b2d49b8af, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680861323209 2023-04-07 09:55:26,427 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting dfdbaa976e8643b0aebf7d4008ebe4fd, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680861323209 2023-04-07 09:55:26,427 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 069114fb05144a0386c42d9079de4346, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1680861323302 2023-04-07 09:55:26,427 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting dc7f0f8d140542b283fd47dbcce8a241, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1680861323302 2023-04-07 09:55:26,428 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b3b653250ff94435a4eb7730d2202290, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680861324441 2023-04-07 09:55:26,428 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 50a7468b0cb04c8aba3b59412d0a6ab5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680861324441 2023-04-07 09:55:26,434 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:26,438 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#431 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:26,447 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/8f9f09f6301e4e06ae9331415352d65f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8f9f09f6301e4e06ae9331415352d65f 2023-04-07 09:55:26,457 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into 8f9f09f6301e4e06ae9331415352d65f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:26,457 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:26,458 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=13, startTime=1680861326425; duration=0sec 2023-04-07 09:55:26,458 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:26,458 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:26,459 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:26,459 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:26,459 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,459 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1f5a1cd7ba354214a30fdb9b41c7925c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20eaf73efe2743d7be546e3d07f33ceb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/849bb459baa34609894f448ce7e5c99a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.4 K 2023-04-07 09:55:26,459 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1f5a1cd7ba354214a30fdb9b41c7925c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680861323209 2023-04-07 09:55:26,459 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 20eaf73efe2743d7be546e3d07f33ceb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1680861323302 2023-04-07 09:55:26,460 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 849bb459baa34609894f448ce7e5c99a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680861324441 2023-04-07 09:55:26,462 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:26,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:26,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:26,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:26,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:26,465 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:26,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:26,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:26,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,467 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:26,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,468 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:26,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:26,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:26,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:26,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:26,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:26,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:26,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:26,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:26,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:26,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:26,476 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-07 09:55:26,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:26,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:26,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:26,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:26,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:26,477 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:26,493 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407d50d1f1c788c4fc3b75f080e755576c8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d50d1f1c788c4fc3b75f080e755576c8 2023-04-07 09:55:26,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:26,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:26,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1680861386754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45146 deadline: 1680861386755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861386757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861386758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861386759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1680861386863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861386863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45138 deadline: 1680861386865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861386866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:26,879 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/1e6a70ebb1f14414b4717d4d909821fd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1e6a70ebb1f14414b4717d4d909821fd 2023-04-07 09:55:26,882 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into 1e6a70ebb1f14414b4717d4d909821fd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:26,882 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:26,882 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=13, startTime=1680861326426; duration=0sec 2023-04-07 09:55:26,882 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:26,901 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=539, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/501178ff82044b54b7cd6a1cb84e18ed 2023-04-07 09:55:26,911 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/ac3b4c0278084cb39750e76befa2570e 2023-04-07 09:55:27,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1680861387070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861387070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861387070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45138 deadline: 1680861387071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,320 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/20d2febf98414ecf81d60c6c68aaae3f 2023-04-07 09:55:27,323 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/501178ff82044b54b7cd6a1cb84e18ed as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/501178ff82044b54b7cd6a1cb84e18ed 2023-04-07 09:55:27,327 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/501178ff82044b54b7cd6a1cb84e18ed, entries=150, sequenceid=539, filesize=25.4 K 2023-04-07 09:55:27,327 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/ac3b4c0278084cb39750e76befa2570e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/ac3b4c0278084cb39750e76befa2570e 2023-04-07 09:55:27,330 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/ac3b4c0278084cb39750e76befa2570e, entries=150, sequenceid=539, filesize=11.9 K 2023-04-07 09:55:27,331 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/20d2febf98414ecf81d60c6c68aaae3f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20d2febf98414ecf81d60c6c68aaae3f 2023-04-07 09:55:27,337 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20d2febf98414ecf81d60c6c68aaae3f, entries=150, sequenceid=539, filesize=11.9 K 2023-04-07 09:55:27,338 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 5aeda58872a183eaf0216fd608afed2c in 862ms, sequenceid=539, compaction requested=false 2023-04-07 09:55:27,338 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:27,338 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:27,338 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:27,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3baf15[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:27,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:27,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,343 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:27,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:27,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:27,343 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,346 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:27,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:27,348 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:27,348 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:27,348 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:27,348 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:27,348 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:27,350 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,350 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,350 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,354 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,354 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,354 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:27,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:27,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:27,354 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,354 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,354 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:27,354 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:27,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:27,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:55:27,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:27,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:27,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:27,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:27,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:27,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:27,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861387401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45138 deadline: 1680861387402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861387403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1680861387405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,454 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:27,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:27,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:27,457 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:27,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:27,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:27,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,459 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,459 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,459 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,459 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:27,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:27,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:27,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:27,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,460 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:27,461 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,461 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:27,461 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:27,461 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:27,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:27,463 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,463 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:27,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:27,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c8b5fe[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:27,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:27,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:27,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:27,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:27,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:27,469 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:27,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:27,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:27,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:27,470 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:27,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:27,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:27,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:27,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,475 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:27,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,476 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:27,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861387510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45138 deadline: 1680861387510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861387510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1680861387512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,576 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:27,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:27,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:27,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:27,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:27,587 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:27,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:27,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:27,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,589 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,590 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,590 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,590 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,590 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:27,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:27,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:27,592 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:27,592 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,592 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:27,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:27,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:27,604 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:27,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:27,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,607 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:27,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d8b3fde[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:27,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:27,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:27,618 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:27,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:27,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:27,626 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:27,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:27,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:27,626 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:27,626 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:27,627 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,627 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,627 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,628 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,630 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,631 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:27,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:27,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,631 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:27,631 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,631 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:27,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:27,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,655 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/627b72cd199e4c6d83ebe36612c8243f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/627b72cd199e4c6d83ebe36612c8243f 2023-04-07 09:55:27,659 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into 627b72cd199e4c6d83ebe36612c8243f(size=26.2 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-07 09:55:27,659 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:27,659 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=13, startTime=1680861326425; duration=1sec 2023-04-07 09:55:27,659 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:27,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45138 deadline: 1680861387717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45176 deadline: 1680861387717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45164 deadline: 1680861387717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45146 deadline: 1680861387717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,731 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:27,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:27,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:27,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:27,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:27,734 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:27,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:27,734 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:27,735 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,735 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,737 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:27,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:27,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:27,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:27,738 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:27,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:27,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:27,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:27,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:27,740 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,740 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,741 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:27,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:27,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61be20f2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:27,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,743 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:27,743 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:27,744 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,744 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:27,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,745 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:27,745 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,745 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:27,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:27,747 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:27,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:27,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:27,748 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:27,748 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:27,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,750 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:27,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,750 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:27,751 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:27,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:27,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,754 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:27,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:27,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:27,754 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:27,754 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:27,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:27,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:27,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45118 deadline: 1680861387763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,791 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304075c5f711abf504ff9b291d9e2aa84a535 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075c5f711abf504ff9b291d9e2aa84a535 2023-04-07 09:55:27,797 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=552, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/1070937f9c544202888a3ea4a876459b 2023-04-07 09:55:27,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/956582bc98a44581a4be0147455bdeb2 2023-04-07 09:55:27,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/447b8000b8ad45bbaf4630a83fb68482 2023-04-07 09:55:27,824 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/1070937f9c544202888a3ea4a876459b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1070937f9c544202888a3ea4a876459b 2023-04-07 09:55:27,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1070937f9c544202888a3ea4a876459b, entries=200, sequenceid=552, filesize=32.3 K 2023-04-07 09:55:27,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/956582bc98a44581a4be0147455bdeb2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/956582bc98a44581a4be0147455bdeb2 2023-04-07 09:55:27,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/956582bc98a44581a4be0147455bdeb2, entries=150, sequenceid=552, filesize=11.9 K 2023-04-07 09:55:27,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/447b8000b8ad45bbaf4630a83fb68482 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/447b8000b8ad45bbaf4630a83fb68482 2023-04-07 09:55:27,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/447b8000b8ad45bbaf4630a83fb68482, entries=150, sequenceid=552, filesize=11.9 K 2023-04-07 09:55:27,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5aeda58872a183eaf0216fd608afed2c in 458ms, sequenceid=552, compaction requested=true 2023-04-07 09:55:27,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:27,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:27,836 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:27,837 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:27,837 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:27,837 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:27,839 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:27,839 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85996 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:27,839 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/B is initiating minor compaction (all files) 2023-04-07 09:55:27,839 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/A is initiating minor compaction (all files) 2023-04-07 09:55:27,839 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/B in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,839 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/A in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,839 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8f9f09f6301e4e06ae9331415352d65f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/ac3b4c0278084cb39750e76befa2570e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/956582bc98a44581a4be0147455bdeb2] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.5 K 2023-04-07 09:55:27,839 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/627b72cd199e4c6d83ebe36612c8243f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/501178ff82044b54b7cd6a1cb84e18ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1070937f9c544202888a3ea4a876459b] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=84.0 K 2023-04-07 09:55:27,839 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8f9f09f6301e4e06ae9331415352d65f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680861324441 2023-04-07 09:55:27,840 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ac3b4c0278084cb39750e76befa2570e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861325618 2023-04-07 09:55:27,841 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 956582bc98a44581a4be0147455bdeb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680861326749 2023-04-07 09:55:27,848 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 627b72cd199e4c6d83ebe36612c8243f, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680861324441 2023-04-07 09:55:27,849 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 501178ff82044b54b7cd6a1cb84e18ed, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861325618 2023-04-07 09:55:27,849 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1070937f9c544202888a3ea4a876459b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680861326749 2023-04-07 09:55:27,855 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:27,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:27,855 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#B#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:27,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:27,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:27,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:27,861 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:27,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:27,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:27,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:27,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:27,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:27,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:27,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:27,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:27,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:27,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:27,872 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:27,872 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:27,872 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:27,876 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:27,876 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:27,876 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,876 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. started... 2023-04-07 09:55:27,876 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:55:27,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:27,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:27,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:27,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:27,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:27,877 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:27,877 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#A#compaction#441 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:27,896 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/492d7c856718429091619dfc470d1270 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/492d7c856718429091619dfc470d1270 2023-04-07 09:55:27,902 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/B of 5aeda58872a183eaf0216fd608afed2c into 492d7c856718429091619dfc470d1270(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:27,903 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:27,903 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/B, priority=13, startTime=1680861327836; duration=0sec 2023-04-07 09:55:27,903 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:27,914 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:27,915 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:27,915 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 5aeda58872a183eaf0216fd608afed2c/C is initiating minor compaction (all files) 2023-04-07 09:55:27,915 DEBUG [Thread-2691] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:49660 2023-04-07 09:55:27,915 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5aeda58872a183eaf0216fd608afed2c/C in TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:27,916 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1e6a70ebb1f14414b4717d4d909821fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20d2febf98414ecf81d60c6c68aaae3f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/447b8000b8ad45bbaf4630a83fb68482] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp, totalSize=36.5 K 2023-04-07 09:55:27,915 DEBUG [Thread-2691] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:27,916 DEBUG [Thread-2685] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:49660 2023-04-07 09:55:27,916 DEBUG [Thread-2687] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:49660 2023-04-07 09:55:27,916 DEBUG [Thread-2687] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:27,916 DEBUG [Thread-2685] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:27,916 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1e6a70ebb1f14414b4717d4d909821fd, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680861324441 2023-04-07 09:55:27,916 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 20d2febf98414ecf81d60c6c68aaae3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1680861325618 2023-04-07 09:55:27,917 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 447b8000b8ad45bbaf4630a83fb68482, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1680861326749 2023-04-07 09:55:27,921 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040755231b9fc46b4a8c910c12fda3a6bf32 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040755231b9fc46b4a8c910c12fda3a6bf32 2023-04-07 09:55:27,922 DEBUG [Thread-2689] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:49660 2023-04-07 09:55:27,922 DEBUG [Thread-2689] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:27,922 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 5aeda58872a183eaf0216fd608afed2c#C#compaction#443 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:27,923 DEBUG [Thread-2693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x159c5e0c to 127.0.0.1:49660 2023-04-07 09:55:27,923 DEBUG [Thread-2693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:28,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:28,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. as already flushing 2023-04-07 09:55:28,022 DEBUG [Thread-2680] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:49660 2023-04-07 09:55:28,022 DEBUG [Thread-2674] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:49660 2023-04-07 09:55:28,022 DEBUG [Thread-2680] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:28,022 DEBUG [Thread-2678] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:49660 2023-04-07 09:55:28,022 DEBUG [Thread-2674] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:28,022 DEBUG [Thread-2678] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:28,023 DEBUG [Thread-2676] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:49660 2023-04-07 09:55:28,023 DEBUG [Thread-2676] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:28,325 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=573, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/296014355eb24a82918611e59c107832 2023-04-07 09:55:28,329 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/6dd806b808f3497da184f1aab4c19813 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/6dd806b808f3497da184f1aab4c19813 2023-04-07 09:55:28,333 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/C of 5aeda58872a183eaf0216fd608afed2c into 6dd806b808f3497da184f1aab4c19813(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:28,333 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:28,333 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/C, priority=13, startTime=1680861327837; duration=0sec 2023-04-07 09:55:28,333 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:28,709 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/be1a0cf9bb784ebc890393a58883eb91 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/be1a0cf9bb784ebc890393a58883eb91 2023-04-07 09:55:28,712 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5aeda58872a183eaf0216fd608afed2c/A of 5aeda58872a183eaf0216fd608afed2c into be1a0cf9bb784ebc890393a58883eb91(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:28,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:28,712 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c., storeName=5aeda58872a183eaf0216fd608afed2c/A, priority=13, startTime=1680861327836; duration=0sec 2023-04-07 09:55:28,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:28,738 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/74b56d8dab06428dbc0f3876d6d24905 2023-04-07 09:55:29,145 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/d309f84fa7a241a390a1b035a514bf90 2023-04-07 09:55:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/296014355eb24a82918611e59c107832 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/296014355eb24a82918611e59c107832 2023-04-07 09:55:29,151 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/296014355eb24a82918611e59c107832, entries=150, sequenceid=573, filesize=25.4 K 2023-04-07 09:55:29,152 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/74b56d8dab06428dbc0f3876d6d24905 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/74b56d8dab06428dbc0f3876d6d24905 2023-04-07 09:55:29,154 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/74b56d8dab06428dbc0f3876d6d24905, entries=150, sequenceid=573, filesize=11.9 K 2023-04-07 09:55:29,155 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/d309f84fa7a241a390a1b035a514bf90 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/d309f84fa7a241a390a1b035a514bf90 2023-04-07 09:55:29,157 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/d309f84fa7a241a390a1b035a514bf90, entries=150, sequenceid=573, filesize=11.9 K 2023-04-07 09:55:29,158 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=26.84 KB/27480 for 5aeda58872a183eaf0216fd608afed2c in 1282ms, sequenceid=573, compaction requested=false 2023-04-07 09:55:29,158 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:29,158 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,160 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273b78d7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:29,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:29,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,163 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:29,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:29,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:29,167 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:29,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:29,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:29,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:29,167 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:29,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:29,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:29,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:29,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:29,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:29,169 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:29,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:29,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:29,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:29,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:29,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:29,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:29,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:29,171 DEBUG [Thread-2684] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:29,171 DEBUG [Thread-2684] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:29,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:29,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:29,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:29,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:29,272 DEBUG [Thread-2684] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:29,776 DEBUG [Thread-2682] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:49660 2023-04-07 09:55:29,776 DEBUG [Thread-2682] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:29,776 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 232 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 1878 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 5629 rows 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 1879 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 5635 rows 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 1868 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 5604 rows 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 1891 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 5664 rows 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 1875 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 5625 rows 2023-04-07 09:55:29,777 INFO [Listener at localhost/39511] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 09:55:29,777 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:49660 2023-04-07 09:55:29,778 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:29,782 INFO [Listener at localhost/39511] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 09:55:29,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 09:55:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:29,787 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861329787"}]},"ts":"1680861329787"} 2023-04-07 09:55:29,788 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 09:55:29,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 09:55:29,792 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 09:55:29,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, UNASSIGN}] 2023-04-07 09:55:29,795 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, UNASSIGN 2023-04-07 09:55:29,795 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:29,795 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861329795"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861329795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861329795"}]},"ts":"1680861329795"} 2023-04-07 09:55:29,796 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 09:55:29,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:29,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5aeda58872a183eaf0216fd608afed2c, disabling compactions & flushes 2023-04-07 09:55:29,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:29,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. after waiting 0 ms 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:29,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5aeda58872a183eaf0216fd608afed2c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=A 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=B 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5aeda58872a183eaf0216fd608afed2c, store=C 2023-04-07 09:55:29,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:30,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 09:55:30,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407bc4ab32c7cb14c6ba39cc2c054a98732 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407bc4ab32c7cb14c6ba39cc2c054a98732 2023-04-07 09:55:30,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 09:55:30,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=584, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/51a5fe3985ca4d9f981d9429eb349e6f 2023-04-07 09:55:30,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 09:55:31,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/274a5c942468406b981f07c488654b05 2023-04-07 09:55:31,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/f8d5d4167e0f4681a82a4e42573f5a4b 2023-04-07 09:55:31,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/A/51a5fe3985ca4d9f981d9429eb349e6f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/51a5fe3985ca4d9f981d9429eb349e6f 2023-04-07 09:55:31,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/51a5fe3985ca4d9f981d9429eb349e6f, entries=100, sequenceid=584, filesize=18.6 K 2023-04-07 09:55:31,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/B/274a5c942468406b981f07c488654b05 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/274a5c942468406b981f07c488654b05 2023-04-07 09:55:31,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/274a5c942468406b981f07c488654b05, entries=100, sequenceid=584, filesize=9.5 K 2023-04-07 09:55:31,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/.tmp/C/f8d5d4167e0f4681a82a4e42573f5a4b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f8d5d4167e0f4681a82a4e42573f5a4b 2023-04-07 09:55:31,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f8d5d4167e0f4681a82a4e42573f5a4b, entries=100, sequenceid=584, filesize=9.5 K 2023-04-07 09:55:31,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 5aeda58872a183eaf0216fd608afed2c in 1643ms, sequenceid=584, compaction requested=true 2023-04-07 09:55:31,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/235ad67f3bc84bf1bd0384927759354d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/b704f6947afe4d9da81dc28959661359, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d331b1dc2c184c4f8122bbc41303a051, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/9c15264b57dc4e84bf1915a32cc3d959, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/e0f26c1715d345479e0e23ca536a3636, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/cbf7c985dede49cfbec87d13b62eb1b8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/11e51a84dcb547f1b678d886ddcad082, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/8e8327c8fffd4a4d9f9b7f85bf81802c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d326bd8736f04f149b6cadd4fa8b48bf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/319d0436743942e4bcbf2f15eed276aa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/34f94ec2a27d453fb81ac6df427d336b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/25ae5d96cd9f4f3388cab4e19384e4a5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4c4de862b0ad43c1bdd4892715d39d3e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/92e57254ee104860b8ba63bacba811ca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/198fb9677a3e49c9ab7aa5af703d4791, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1d22056991a041359e492553c3e87239, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4f733dfcb0f741ec9c909bdba32a5a0a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/80ac3f86811d4c3ab499d87c0c52dc61, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/0f6abe398b9b485fb9f4473c0057d9ec, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4abfef2f8fd041f88c7be578e9cbe321, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/13763ac236de46fe8fdefefebc42de4a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dfdbaa976e8643b0aebf7d4008ebe4fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dc7f0f8d140542b283fd47dbcce8a241, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/50a7468b0cb04c8aba3b59412d0a6ab5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/627b72cd199e4c6d83ebe36612c8243f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/501178ff82044b54b7cd6a1cb84e18ed, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1070937f9c544202888a3ea4a876459b] to archive 2023-04-07 09:55:31,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:31,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/235ad67f3bc84bf1bd0384927759354d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/235ad67f3bc84bf1bd0384927759354d 2023-04-07 09:55:31,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/b704f6947afe4d9da81dc28959661359 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/b704f6947afe4d9da81dc28959661359 2023-04-07 09:55:31,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d331b1dc2c184c4f8122bbc41303a051 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d331b1dc2c184c4f8122bbc41303a051 2023-04-07 09:55:31,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/9c15264b57dc4e84bf1915a32cc3d959 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/9c15264b57dc4e84bf1915a32cc3d959 2023-04-07 09:55:31,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/e0f26c1715d345479e0e23ca536a3636 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/e0f26c1715d345479e0e23ca536a3636 2023-04-07 09:55:31,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/cbf7c985dede49cfbec87d13b62eb1b8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/cbf7c985dede49cfbec87d13b62eb1b8 2023-04-07 09:55:31,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/11e51a84dcb547f1b678d886ddcad082 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/11e51a84dcb547f1b678d886ddcad082 2023-04-07 09:55:31,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/8e8327c8fffd4a4d9f9b7f85bf81802c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/8e8327c8fffd4a4d9f9b7f85bf81802c 2023-04-07 09:55:31,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d326bd8736f04f149b6cadd4fa8b48bf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/d326bd8736f04f149b6cadd4fa8b48bf 2023-04-07 09:55:31,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/319d0436743942e4bcbf2f15eed276aa to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/319d0436743942e4bcbf2f15eed276aa 2023-04-07 09:55:31,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/34f94ec2a27d453fb81ac6df427d336b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/34f94ec2a27d453fb81ac6df427d336b 2023-04-07 09:55:31,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/25ae5d96cd9f4f3388cab4e19384e4a5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/25ae5d96cd9f4f3388cab4e19384e4a5 2023-04-07 09:55:31,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4c4de862b0ad43c1bdd4892715d39d3e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4c4de862b0ad43c1bdd4892715d39d3e 2023-04-07 09:55:31,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/92e57254ee104860b8ba63bacba811ca to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/92e57254ee104860b8ba63bacba811ca 2023-04-07 09:55:31,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/198fb9677a3e49c9ab7aa5af703d4791 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/198fb9677a3e49c9ab7aa5af703d4791 2023-04-07 09:55:31,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1d22056991a041359e492553c3e87239 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1d22056991a041359e492553c3e87239 2023-04-07 09:55:31,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4f733dfcb0f741ec9c909bdba32a5a0a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4f733dfcb0f741ec9c909bdba32a5a0a 2023-04-07 09:55:31,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/80ac3f86811d4c3ab499d87c0c52dc61 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/80ac3f86811d4c3ab499d87c0c52dc61 2023-04-07 09:55:31,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/0f6abe398b9b485fb9f4473c0057d9ec to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/0f6abe398b9b485fb9f4473c0057d9ec 2023-04-07 09:55:31,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4abfef2f8fd041f88c7be578e9cbe321 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/4abfef2f8fd041f88c7be578e9cbe321 2023-04-07 09:55:31,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/13763ac236de46fe8fdefefebc42de4a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/13763ac236de46fe8fdefefebc42de4a 2023-04-07 09:55:31,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dfdbaa976e8643b0aebf7d4008ebe4fd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dfdbaa976e8643b0aebf7d4008ebe4fd 2023-04-07 09:55:31,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dc7f0f8d140542b283fd47dbcce8a241 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/dc7f0f8d140542b283fd47dbcce8a241 2023-04-07 09:55:31,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/50a7468b0cb04c8aba3b59412d0a6ab5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/50a7468b0cb04c8aba3b59412d0a6ab5 2023-04-07 09:55:31,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/627b72cd199e4c6d83ebe36612c8243f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/627b72cd199e4c6d83ebe36612c8243f 2023-04-07 09:55:31,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/501178ff82044b54b7cd6a1cb84e18ed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/501178ff82044b54b7cd6a1cb84e18ed 2023-04-07 09:55:31,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1070937f9c544202888a3ea4a876459b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/1070937f9c544202888a3ea4a876459b 2023-04-07 09:55:31,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/82d0ed8ac6c5466d9e40a9a43c11930d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/12c5a4b9824f4e01ae8b0c9bc39d335b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/da4bbc22e9864acc9905baf920d93cf5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/39a3c0f97dc24a64a2ed404220f3adac, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/86068b6bf75247c887557317b093f0ca, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/1091efb96fb64d2b9e5cbe70ad2d751b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/3d7fb27a4673454c8051a1b721b3fc8f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cdc2c2fe1ced48cbae7a78c4de84edd5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/42baac8374ac45b8b50fdf9a4c752c28, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8e226cf307f9435791f096a8d33dbeb2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/fb304b66fc3b4f51958eb082fdd895c5, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28c0b0ae3fce44a6b385c46b50d4e6c1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/daa5a251b98943f2ba59fcd21beca2fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/353f6a021492409faeda6e71570d3e89, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cca54487be2848f5b2009d81544c84bc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28df293b92214e4a8cb207b2c179f636, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8af65f739c7e4d3db4edde9fbca80f7e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/6030cd46ae074333a4da8ca9e9012cba, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/101828e0845c4749a1d52fd3a9d6d7bc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cd99444003714124adb37c5943f0e32b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b30879dfdd854e1d99865a2b2d49b8af, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/4e249a939e2a416b8e557155fd09e077, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/069114fb05144a0386c42d9079de4346, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8f9f09f6301e4e06ae9331415352d65f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b3b653250ff94435a4eb7730d2202290, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/ac3b4c0278084cb39750e76befa2570e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/956582bc98a44581a4be0147455bdeb2] to archive 2023-04-07 09:55:31,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:31,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/82d0ed8ac6c5466d9e40a9a43c11930d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/82d0ed8ac6c5466d9e40a9a43c11930d 2023-04-07 09:55:31,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/12c5a4b9824f4e01ae8b0c9bc39d335b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/12c5a4b9824f4e01ae8b0c9bc39d335b 2023-04-07 09:55:31,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/da4bbc22e9864acc9905baf920d93cf5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/da4bbc22e9864acc9905baf920d93cf5 2023-04-07 09:55:31,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/39a3c0f97dc24a64a2ed404220f3adac to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/39a3c0f97dc24a64a2ed404220f3adac 2023-04-07 09:55:31,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/86068b6bf75247c887557317b093f0ca to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/86068b6bf75247c887557317b093f0ca 2023-04-07 09:55:31,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/1091efb96fb64d2b9e5cbe70ad2d751b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/1091efb96fb64d2b9e5cbe70ad2d751b 2023-04-07 09:55:31,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/3d7fb27a4673454c8051a1b721b3fc8f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/3d7fb27a4673454c8051a1b721b3fc8f 2023-04-07 09:55:31,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cdc2c2fe1ced48cbae7a78c4de84edd5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cdc2c2fe1ced48cbae7a78c4de84edd5 2023-04-07 09:55:31,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/42baac8374ac45b8b50fdf9a4c752c28 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/42baac8374ac45b8b50fdf9a4c752c28 2023-04-07 09:55:31,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8e226cf307f9435791f096a8d33dbeb2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8e226cf307f9435791f096a8d33dbeb2 2023-04-07 09:55:31,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/fb304b66fc3b4f51958eb082fdd895c5 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/fb304b66fc3b4f51958eb082fdd895c5 2023-04-07 09:55:31,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28c0b0ae3fce44a6b385c46b50d4e6c1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28c0b0ae3fce44a6b385c46b50d4e6c1 2023-04-07 09:55:31,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/daa5a251b98943f2ba59fcd21beca2fd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/daa5a251b98943f2ba59fcd21beca2fd 2023-04-07 09:55:31,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/353f6a021492409faeda6e71570d3e89 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/353f6a021492409faeda6e71570d3e89 2023-04-07 09:55:31,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cca54487be2848f5b2009d81544c84bc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cca54487be2848f5b2009d81544c84bc 2023-04-07 09:55:31,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28df293b92214e4a8cb207b2c179f636 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/28df293b92214e4a8cb207b2c179f636 2023-04-07 09:55:31,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8af65f739c7e4d3db4edde9fbca80f7e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8af65f739c7e4d3db4edde9fbca80f7e 2023-04-07 09:55:31,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/6030cd46ae074333a4da8ca9e9012cba to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/6030cd46ae074333a4da8ca9e9012cba 2023-04-07 09:55:31,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/101828e0845c4749a1d52fd3a9d6d7bc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/101828e0845c4749a1d52fd3a9d6d7bc 2023-04-07 09:55:31,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cd99444003714124adb37c5943f0e32b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/cd99444003714124adb37c5943f0e32b 2023-04-07 09:55:31,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b30879dfdd854e1d99865a2b2d49b8af to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b30879dfdd854e1d99865a2b2d49b8af 2023-04-07 09:55:31,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/4e249a939e2a416b8e557155fd09e077 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/4e249a939e2a416b8e557155fd09e077 2023-04-07 09:55:31,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/069114fb05144a0386c42d9079de4346 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/069114fb05144a0386c42d9079de4346 2023-04-07 09:55:31,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8f9f09f6301e4e06ae9331415352d65f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/8f9f09f6301e4e06ae9331415352d65f 2023-04-07 09:55:31,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b3b653250ff94435a4eb7730d2202290 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/b3b653250ff94435a4eb7730d2202290 2023-04-07 09:55:31,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/ac3b4c0278084cb39750e76befa2570e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/ac3b4c0278084cb39750e76befa2570e 2023-04-07 09:55:31,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/956582bc98a44581a4be0147455bdeb2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/956582bc98a44581a4be0147455bdeb2 2023-04-07 09:55:31,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1b8afb2f849748bab28858b5d7e4f83f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a8d2bb7d9ecd4d7f89c2455250eef463, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a3ac8b523c20408494d414e4811d822b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/c19c309418814d1c87900f5b02bdf22a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/8b7ae1c6a41945838f615ff22338bfbd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fdeccb4dd55b41b29267010e95edc912, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/7917ad1e41bf45d78983781f800eeb74, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/b0c0260b53734d0e8d686b1cf1b761b8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e977350673894b0c95f0af3732ad2bda, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/0460ef743fbd489681afa3ddf555c026, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fc8911494277452882edf57d56fefbdf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/511eb425af234404a029bf5bae1387f8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/11151e2dbf4445f791df109c8a13757b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f55d4adec0f146b7b22908fc4387485e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e6b1fd30c87545218ea2160d432a3dbe, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/28c9064b96df4a2eb0ea32bde7e60d9c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/468f2cbbc80e4438942b5fd6e16f3e32, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/578317b0997c407ba680bbb6495bc7fa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e9f7f79c0261426a8719e08f22777551, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/30a25c7002294794a86627feda000f5f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1f5a1cd7ba354214a30fdb9b41c7925c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/56d9551739aa4dd097611dd666fe96a6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20eaf73efe2743d7be546e3d07f33ceb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1e6a70ebb1f14414b4717d4d909821fd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/849bb459baa34609894f448ce7e5c99a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20d2febf98414ecf81d60c6c68aaae3f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/447b8000b8ad45bbaf4630a83fb68482] to archive 2023-04-07 09:55:31,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:31,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1b8afb2f849748bab28858b5d7e4f83f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1b8afb2f849748bab28858b5d7e4f83f 2023-04-07 09:55:31,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a8d2bb7d9ecd4d7f89c2455250eef463 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a8d2bb7d9ecd4d7f89c2455250eef463 2023-04-07 09:55:31,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a3ac8b523c20408494d414e4811d822b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/a3ac8b523c20408494d414e4811d822b 2023-04-07 09:55:31,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/c19c309418814d1c87900f5b02bdf22a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/c19c309418814d1c87900f5b02bdf22a 2023-04-07 09:55:31,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/8b7ae1c6a41945838f615ff22338bfbd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/8b7ae1c6a41945838f615ff22338bfbd 2023-04-07 09:55:31,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fdeccb4dd55b41b29267010e95edc912 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fdeccb4dd55b41b29267010e95edc912 2023-04-07 09:55:31,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/7917ad1e41bf45d78983781f800eeb74 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/7917ad1e41bf45d78983781f800eeb74 2023-04-07 09:55:31,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/b0c0260b53734d0e8d686b1cf1b761b8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/b0c0260b53734d0e8d686b1cf1b761b8 2023-04-07 09:55:31,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e977350673894b0c95f0af3732ad2bda to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e977350673894b0c95f0af3732ad2bda 2023-04-07 09:55:31,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/0460ef743fbd489681afa3ddf555c026 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/0460ef743fbd489681afa3ddf555c026 2023-04-07 09:55:31,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fc8911494277452882edf57d56fefbdf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/fc8911494277452882edf57d56fefbdf 2023-04-07 09:55:31,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/511eb425af234404a029bf5bae1387f8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/511eb425af234404a029bf5bae1387f8 2023-04-07 09:55:31,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/11151e2dbf4445f791df109c8a13757b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/11151e2dbf4445f791df109c8a13757b 2023-04-07 09:55:31,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f55d4adec0f146b7b22908fc4387485e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f55d4adec0f146b7b22908fc4387485e 2023-04-07 09:55:31,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e6b1fd30c87545218ea2160d432a3dbe to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e6b1fd30c87545218ea2160d432a3dbe 2023-04-07 09:55:31,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/28c9064b96df4a2eb0ea32bde7e60d9c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/28c9064b96df4a2eb0ea32bde7e60d9c 2023-04-07 09:55:31,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/468f2cbbc80e4438942b5fd6e16f3e32 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/468f2cbbc80e4438942b5fd6e16f3e32 2023-04-07 09:55:31,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/578317b0997c407ba680bbb6495bc7fa to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/578317b0997c407ba680bbb6495bc7fa 2023-04-07 09:55:31,677 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e9f7f79c0261426a8719e08f22777551 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/e9f7f79c0261426a8719e08f22777551 2023-04-07 09:55:31,678 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/30a25c7002294794a86627feda000f5f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/30a25c7002294794a86627feda000f5f 2023-04-07 09:55:31,678 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1f5a1cd7ba354214a30fdb9b41c7925c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1f5a1cd7ba354214a30fdb9b41c7925c 2023-04-07 09:55:31,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/56d9551739aa4dd097611dd666fe96a6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/56d9551739aa4dd097611dd666fe96a6 2023-04-07 09:55:31,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20eaf73efe2743d7be546e3d07f33ceb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20eaf73efe2743d7be546e3d07f33ceb 2023-04-07 09:55:31,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1e6a70ebb1f14414b4717d4d909821fd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/1e6a70ebb1f14414b4717d4d909821fd 2023-04-07 09:55:31,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/849bb459baa34609894f448ce7e5c99a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/849bb459baa34609894f448ce7e5c99a 2023-04-07 09:55:31,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20d2febf98414ecf81d60c6c68aaae3f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/20d2febf98414ecf81d60c6c68aaae3f 2023-04-07 09:55:31,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/447b8000b8ad45bbaf4630a83fb68482 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/447b8000b8ad45bbaf4630a83fb68482 2023-04-07 09:55:31,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/recovered.edits/587.seqid, newMaxSeqId=587, maxSeqId=4 2023-04-07 09:55:31,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c. 2023-04-07 09:55:31,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5aeda58872a183eaf0216fd608afed2c: 2023-04-07 09:55:31,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:31,688 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5aeda58872a183eaf0216fd608afed2c, regionState=CLOSED 2023-04-07 09:55:31,689 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861331688"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861331688"}]},"ts":"1680861331688"} 2023-04-07 09:55:31,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-07 09:55:31,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 5aeda58872a183eaf0216fd608afed2c, server=jenkins-hbase4.apache.org,35881,1680861218288 in 1.8930 sec 2023-04-07 09:55:31,691 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-07 09:55:31,691 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5aeda58872a183eaf0216fd608afed2c, UNASSIGN in 1.8970 sec 2023-04-07 09:55:31,692 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861331692"}]},"ts":"1680861331692"} 2023-04-07 09:55:31,692 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 09:55:31,694 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 09:55:31,695 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9110 sec 2023-04-07 09:55:31,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-07 09:55:31,892 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-07 09:55:31,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 09:55:31,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:31,895 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:31,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-07 09:55:31,896 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:31,899 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:31,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/recovered.edits] 2023-04-07 09:55:31,903 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/296014355eb24a82918611e59c107832 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/296014355eb24a82918611e59c107832 2023-04-07 09:55:31,904 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/51a5fe3985ca4d9f981d9429eb349e6f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/51a5fe3985ca4d9f981d9429eb349e6f 2023-04-07 09:55:31,905 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/be1a0cf9bb784ebc890393a58883eb91 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/A/be1a0cf9bb784ebc890393a58883eb91 2023-04-07 09:55:31,907 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/274a5c942468406b981f07c488654b05 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/274a5c942468406b981f07c488654b05 2023-04-07 09:55:31,908 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/492d7c856718429091619dfc470d1270 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/492d7c856718429091619dfc470d1270 2023-04-07 09:55:31,909 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/74b56d8dab06428dbc0f3876d6d24905 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/B/74b56d8dab06428dbc0f3876d6d24905 2023-04-07 09:55:31,910 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/6dd806b808f3497da184f1aab4c19813 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/6dd806b808f3497da184f1aab4c19813 2023-04-07 09:55:31,912 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/d309f84fa7a241a390a1b035a514bf90 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/d309f84fa7a241a390a1b035a514bf90 2023-04-07 09:55:31,913 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f8d5d4167e0f4681a82a4e42573f5a4b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/C/f8d5d4167e0f4681a82a4e42573f5a4b 2023-04-07 09:55:31,915 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/recovered.edits/587.seqid to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c/recovered.edits/587.seqid 2023-04-07 09:55:31,916 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/5aeda58872a183eaf0216fd608afed2c 2023-04-07 09:55:31,916 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:55:31,916 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 09:55:31,917 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-07 09:55:31,920 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407061ef14588224fc68bb41be403174341 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407061ef14588224fc68bb41be403174341 2023-04-07 09:55:31,921 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407155b8e73ea13410ea19b0598d3462c88 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407155b8e73ea13410ea19b0598d3462c88 2023-04-07 09:55:31,922 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073c5c2f147111421ba2fcdb88b5e62028 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073c5c2f147111421ba2fcdb88b5e62028 2023-04-07 09:55:31,923 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073e50b1b57df6423dac36c99bbe033c3c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304073e50b1b57df6423dac36c99bbe033c3c 2023-04-07 09:55:31,924 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407402a5dd03b344bca8de1d265746076df to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407402a5dd03b344bca8de1d265746076df 2023-04-07 09:55:31,927 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040740c26cc143c047ea93dda8cd5ff75859 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040740c26cc143c047ea93dda8cd5ff75859 2023-04-07 09:55:31,929 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074bae907952c74770827413f6bb7990ec to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304074bae907952c74770827413f6bb7990ec 2023-04-07 09:55:31,930 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040755231b9fc46b4a8c910c12fda3a6bf32 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040755231b9fc46b4a8c910c12fda3a6bf32 2023-04-07 09:55:31,931 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040758ea43d4c00f4a42bcd94a561f204ca7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040758ea43d4c00f4a42bcd94a561f204ca7 2023-04-07 09:55:31,932 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075ac58e959734406bac0c7ffc11c666c7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075ac58e959734406bac0c7ffc11c666c7 2023-04-07 09:55:31,934 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075c5f711abf504ff9b291d9e2aa84a535 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304075c5f711abf504ff9b291d9e2aa84a535 2023-04-07 09:55:31,935 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040768055a4e0e6748289811d18bf1d52329 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040768055a4e0e6748289811d18bf1d52329 2023-04-07 09:55:31,936 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407762dfa15021243bb915ecd2b8d243eed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407762dfa15021243bb915ecd2b8d243eed 2023-04-07 09:55:31,938 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a842fb5d5bdc45e89c3f8b9660ee2045 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a842fb5d5bdc45e89c3f8b9660ee2045 2023-04-07 09:55:31,939 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b72093faa6cd4e5f931b25ec3d754ffd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b72093faa6cd4e5f931b25ec3d754ffd 2023-04-07 09:55:31,940 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407bc4ab32c7cb14c6ba39cc2c054a98732 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407bc4ab32c7cb14c6ba39cc2c054a98732 2023-04-07 09:55:31,942 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d22b891820aa4710bd8cce497d3a0a13 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d22b891820aa4710bd8cce497d3a0a13 2023-04-07 09:55:31,944 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d36adbe039714199a205b4c4104f4090 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d36adbe039714199a205b4c4104f4090 2023-04-07 09:55:31,945 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d50d1f1c788c4fc3b75f080e755576c8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d50d1f1c788c4fc3b75f080e755576c8 2023-04-07 09:55:31,947 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e6bab95b886b4a06b95b77ab9ded71e3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e6bab95b886b4a06b95b77ab9ded71e3 2023-04-07 09:55:31,948 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f4529d4a84ee4c3597b9ab8ae26df0ac to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f4529d4a84ee4c3597b9ab8ae26df0ac 2023-04-07 09:55:31,949 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f7c47d40ea854ab9a7eaf1d88bf5233f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f7c47d40ea854ab9a7eaf1d88bf5233f 2023-04-07 09:55:31,950 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 09:55:31,957 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:31,959 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 09:55:31,960 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 09:55:31,961 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:31,961 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 09:55:31,962 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680861331961"}]},"ts":"9223372036854775807"} 2023-04-07 09:55:31,966 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 09:55:31,966 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5aeda58872a183eaf0216fd608afed2c, NAME => 'TestAcidGuarantees,,1680861304831.5aeda58872a183eaf0216fd608afed2c.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 09:55:31,966 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 09:55:31,966 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680861331966"}]},"ts":"9223372036854775807"} 2023-04-07 09:55:31,971 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 09:55:31,973 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:31,974 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 80 msec 2023-04-07 09:55:31,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-07 09:55:31,996 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-07 09:55:32,010 INFO [Listener at localhost/39511] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 229), OpenFileDescriptor=554 (was 590), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=453 (was 486), ProcessCount=169 (was 169), AvailableMemoryMB=2563 (was 2712) 2023-04-07 09:55:32,024 INFO [Listener at localhost/39511] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=554, MaxFileDescriptor=60000, SystemLoadAverage=453, ProcessCount=169, AvailableMemoryMB=2563 2023-04-07 09:55:32,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:55:32,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:55:32,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:32,029 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 09:55:32,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-07 09:55:32,029 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 09:55:32,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 09:55:32,030 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:32,031 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2 empty. 2023-04-07 09:55:32,031 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:32,031 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:55:32,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 09:55:32,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 09:55:32,437 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:55:32,439 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c05568a99c8bbb1b0521e425f41a54f2, NAME => 'TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp 2023-04-07 09:55:32,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 09:55:32,844 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:55:32,844 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c05568a99c8bbb1b0521e425f41a54f2, disabling compactions & flushes 2023-04-07 09:55:32,844 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:32,844 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:32,844 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. after waiting 0 ms 2023-04-07 09:55:32,844 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:32,844 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:32,844 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:32,846 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 09:55:32,847 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861332847"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861332847"}]},"ts":"1680861332847"} 2023-04-07 09:55:32,848 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 09:55:32,848 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 09:55:32,849 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861332849"}]},"ts":"1680861332849"} 2023-04-07 09:55:32,849 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 09:55:32,853 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c05568a99c8bbb1b0521e425f41a54f2, ASSIGN}] 2023-04-07 09:55:32,854 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c05568a99c8bbb1b0521e425f41a54f2, ASSIGN 2023-04-07 09:55:32,854 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c05568a99c8bbb1b0521e425f41a54f2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=false 2023-04-07 09:55:33,006 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=c05568a99c8bbb1b0521e425f41a54f2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:33,006 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861333006"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861333006"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861333006"}]},"ts":"1680861333006"} 2023-04-07 09:55:33,007 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:33,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 09:55:33,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:33,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c05568a99c8bbb1b0521e425f41a54f2, NAME => 'TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:55:33,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:55:33,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,163 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,164 DEBUG [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A 2023-04-07 09:55:33,164 DEBUG [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A 2023-04-07 09:55:33,164 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:33,164 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c05568a99c8bbb1b0521e425f41a54f2 columnFamilyName A 2023-04-07 09:55:33,165 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.HStore(310): Store=c05568a99c8bbb1b0521e425f41a54f2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:33,165 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,166 DEBUG [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B 2023-04-07 09:55:33,166 DEBUG [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B 2023-04-07 09:55:33,166 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:33,166 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c05568a99c8bbb1b0521e425f41a54f2 columnFamilyName B 2023-04-07 09:55:33,166 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.HStore(310): Store=c05568a99c8bbb1b0521e425f41a54f2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:33,166 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,167 DEBUG [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C 2023-04-07 09:55:33,167 DEBUG [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C 2023-04-07 09:55:33,168 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:33,168 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c05568a99c8bbb1b0521e425f41a54f2 columnFamilyName C 2023-04-07 09:55:33,168 INFO [StoreOpener-c05568a99c8bbb1b0521e425f41a54f2-1] regionserver.HStore(310): Store=c05568a99c8bbb1b0521e425f41a54f2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:33,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:33,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:55:33,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:33,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:55:33,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c05568a99c8bbb1b0521e425f41a54f2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67590039, jitterRate=0.007170066237449646}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:55:33,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:33,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., pid=49, masterSystemTime=1680861333158 2023-04-07 09:55:33,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:33,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:33,175 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=c05568a99c8bbb1b0521e425f41a54f2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:33,175 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861333175"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861333175"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861333175"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861333175"}]},"ts":"1680861333175"} 2023-04-07 09:55:33,177 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-07 09:55:33,177 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 in 169 msec 2023-04-07 09:55:33,178 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-07 09:55:33,178 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c05568a99c8bbb1b0521e425f41a54f2, ASSIGN in 324 msec 2023-04-07 09:55:33,179 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 09:55:33,179 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861333179"}]},"ts":"1680861333179"} 2023-04-07 09:55:33,179 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 09:55:33,181 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 09:55:33,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1540 sec 2023-04-07 09:55:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-07 09:55:34,134 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-07 09:55:34,135 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,140 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,141 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,143 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,144 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 09:55:34,145 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45944, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 09:55:34,146 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,150 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,150 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,156 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,156 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,160 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,161 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,166 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,166 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,170 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,170 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,176 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,176 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,181 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,181 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,184 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,184 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,191 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,191 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x4ac98310 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:34,195 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a9f3972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:34,205 DEBUG [hconnection-0x7f8d4523-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,205 DEBUG [hconnection-0x65b44426-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,205 DEBUG [hconnection-0x31041e5d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,206 DEBUG [hconnection-0x6853fa73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,206 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,206 DEBUG [hconnection-0x676cfbde-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,207 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,208 DEBUG [hconnection-0x439721e4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,208 DEBUG [hconnection-0x6c21532-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,212 DEBUG [hconnection-0x27681fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,214 DEBUG [hconnection-0x43e279bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,215 DEBUG [hconnection-0x21364e85-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:34,217 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49544, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,219 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,219 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,219 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,219 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,220 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,220 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49554, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,227 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:34,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:34,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:34,237 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:55:34,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:34,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:34,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:34,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:34,244 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:34,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:34,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:34,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,246 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,246 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:34,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:34,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:34,247 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:34,247 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,247 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:34,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:34,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:34,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861394245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861394245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861394246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861394247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861394247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:34,262 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,262 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,263 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24df5e8b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:34,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:34,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,265 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:34,265 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:34,265 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,265 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:34,267 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:34,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:34,270 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:34,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:34,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:34,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:34,270 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:34,271 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,271 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,271 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,272 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,275 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:34,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,275 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:34,275 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:34,275 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:34,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861394352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861394352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861394353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861394354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861394355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,375 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:34,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:34,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:34,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:34,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:34,386 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:34,386 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,386 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:34,386 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:34,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,387 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,388 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,388 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:34,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:34,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:34,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:34,389 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:34,389 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,389 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:34,389 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:34,389 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:34,390 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:34,390 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:34,390 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:34,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:34,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:34,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:34,392 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,392 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,392 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:34,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:34,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8ef001[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:34,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,395 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:34,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:34,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:34,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:34,396 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:34,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:34,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:34,399 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:34,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:34,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:34,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:34,399 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:34,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,403 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,405 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,405 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,405 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:34,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:34,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,406 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:34,406 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:34,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,506 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:34,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:34,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:34,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:34,510 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:34,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:34,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:34,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,510 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,511 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,512 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:34,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:34,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:34,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:34,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:34,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:34,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:34,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:34,517 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:34,517 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:34,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:34,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:34,517 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:34,518 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,518 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,518 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,519 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:34,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1027e952[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:34,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:34,520 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,526 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,526 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:34,526 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:34,526 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:34,526 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,526 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:34,528 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:34,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:34,530 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:34,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:34,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:34,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:34,530 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:34,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,532 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,534 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:34,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:34,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:34,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,536 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,537 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,537 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,537 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:34,537 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:34,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861394555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861394556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861394557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861394558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861394560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,637 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:34,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:34,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:34,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:34,640 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:34,640 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,640 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:34,640 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:34,640 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,640 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,643 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:34,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:34,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:34,645 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,645 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:34,645 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:34,645 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:34,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:34,647 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:34,650 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:34,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c26c641[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:34,650 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,651 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:34,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:34,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:34,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:34,653 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:34,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:34,657 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:34,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:34,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:34,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:34,657 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:34,659 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,659 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,659 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,660 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,661 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/6d70dc3d333a4dc0896aee19a3920db1 2023-04-07 09:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:34,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:34,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:34,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,665 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:34,665 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:34,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/c777fbedb08e491dabe29339e97a0c51 2023-04-07 09:55:34,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/84447971fc6f40ff8e67c6f4d4d0cc97 2023-04-07 09:55:34,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/6d70dc3d333a4dc0896aee19a3920db1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6d70dc3d333a4dc0896aee19a3920db1 2023-04-07 09:55:34,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6d70dc3d333a4dc0896aee19a3920db1, entries=150, sequenceid=14, filesize=11.6 K 2023-04-07 09:55:34,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/c777fbedb08e491dabe29339e97a0c51 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c777fbedb08e491dabe29339e97a0c51 2023-04-07 09:55:34,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c777fbedb08e491dabe29339e97a0c51, entries=100, sequenceid=14, filesize=9.3 K 2023-04-07 09:55:34,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/84447971fc6f40ff8e67c6f4d4d0cc97 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/84447971fc6f40ff8e67c6f4d4d0cc97 2023-04-07 09:55:34,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/84447971fc6f40ff8e67c6f4d4d0cc97, entries=100, sequenceid=14, filesize=9.3 K 2023-04-07 09:55:34,703 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c05568a99c8bbb1b0521e425f41a54f2 in 466ms, sequenceid=14, compaction requested=false 2023-04-07 09:55:34,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:34,765 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:34,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:34,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:34,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:34,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:34,768 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:34,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:34,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:34,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:34,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,771 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:34,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:34,772 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,789 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/bc06d999145145a387d18a6de173f38e 2023-04-07 09:55:34,804 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/deb109c6d6c5459e85fb03f9d8d39cf8 2023-04-07 09:55:34,814 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2b846a3bd91d4626828c552c72d05e81 2023-04-07 09:55:34,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/bc06d999145145a387d18a6de173f38e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc06d999145145a387d18a6de173f38e 2023-04-07 09:55:34,820 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc06d999145145a387d18a6de173f38e, entries=150, sequenceid=33, filesize=11.6 K 2023-04-07 09:55:34,821 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/deb109c6d6c5459e85fb03f9d8d39cf8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/deb109c6d6c5459e85fb03f9d8d39cf8 2023-04-07 09:55:34,826 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/deb109c6d6c5459e85fb03f9d8d39cf8, entries=150, sequenceid=33, filesize=11.6 K 2023-04-07 09:55:34,827 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2b846a3bd91d4626828c552c72d05e81 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2b846a3bd91d4626828c552c72d05e81 2023-04-07 09:55:34,831 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2b846a3bd91d4626828c552c72d05e81, entries=150, sequenceid=33, filesize=11.6 K 2023-04-07 09:55:34,832 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for c05568a99c8bbb1b0521e425f41a54f2 in 59ms, sequenceid=33, compaction requested=false 2023-04-07 09:55:34,832 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:34,832 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:34,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:34,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:34,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:34,833 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3253e648[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:34,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:34,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:34,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:34,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:34,840 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,840 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:34,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:34,845 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:34,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:34,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:34,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:34,845 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:34,847 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,847 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,847 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,848 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,851 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,851 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:34,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:34,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:34,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,852 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:34,852 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:34,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:34,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:34,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:34,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:34,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,866 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:34,866 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861334882, max timestamp=1680861334895Num uniques 859; 2023-04-07 09:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861334882, max timestamp=1680861334895Num uniques 859; 2023-04-07 09:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:34,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861334882, max timestamp=1680861334896Num uniques 859; 2023-04-07 09:55:34,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:34,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861394919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861394919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861394920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861394925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:34,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861394926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,952 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:34,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:34,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:34,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:34,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:34,955 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:34,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:34,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:34,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:34,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:34,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:34,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:34,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:34,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:34,961 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:34,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:34,962 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45aa16fb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:34,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:34,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,965 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:34,965 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,970 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:34,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:34,973 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:34,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:34,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:34,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:34,973 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:34,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,975 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:34,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:34,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:34,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:34,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:34,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:34,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:34,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:34,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:34,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:34,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:34,979 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:34,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:34,979 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:35,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861395028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861395028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861395028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861395032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861395035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,079 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:35,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:35,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:35,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:35,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:35,083 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:35,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:35,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:35,084 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,084 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,085 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,085 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,085 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,085 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:35,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:35,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:35,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:35,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,086 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:35,087 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,087 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:35,087 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:35,087 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:35,088 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,088 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,089 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:35,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a681a65[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:35,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:35,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,095 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,096 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:35,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:35,099 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:35,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:35,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:35,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:35,099 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:35,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,101 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:35,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:35,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,107 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,107 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:35,107 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:35,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:35,207 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:35,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:35,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:35,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:35,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:35,210 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:35,210 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,210 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:35,210 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:35,211 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,211 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,213 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,213 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,213 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,213 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:35,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:35,215 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:35,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:35,216 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:35,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:35,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3026cda7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:35,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:35,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:35,231 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:35,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:35,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:35,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:35,231 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:35,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,235 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,238 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:35,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,240 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:35,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:35,241 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:35,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861395236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861395236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861395236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861395238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861395239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/eac473439da544fba97fdab800d66518 2023-04-07 09:55:35,341 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:35,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:35,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:35,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:35,344 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:35,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:35,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:35,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,345 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,346 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:35,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:35,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:35,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:35,347 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,347 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:35,347 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:35,347 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:35,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:35,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:35,349 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,349 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,349 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,349 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:35,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dea0c4d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:35,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:35,352 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,354 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:35,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,354 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:35,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:35,358 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:35,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:35,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:35,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:35,358 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:35,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,360 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,366 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:35,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:35,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:35,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,367 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,367 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:35,467 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:35,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:35,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:35,470 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:35,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:35,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:35,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,472 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:35,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:35,473 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,473 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:35,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:35,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:35,475 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:35,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@862ccc5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:35,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,479 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:35,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:35,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:35,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:35,483 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:35,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:35,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:35,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:35,483 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:35,485 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,485 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,485 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:35,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,489 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,489 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:35,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,489 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:35,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:35,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861395546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861395547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861395547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861395548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:35,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861395548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,589 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:35,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:35,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:35,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:35,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:35,592 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:35,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:35,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:35,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,599 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:35,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:35,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:35,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:35,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:35,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:35,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:35,602 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:35,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f3449e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:35,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:35,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:35,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:35,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:35,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:35,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,607 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,607 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,607 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:35,609 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:35,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:35,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:35,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:35,609 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:35,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:35,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:35,613 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,615 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,615 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,615 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:35,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:35,615 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:35,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,615 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:35,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:35,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:35,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:35,616 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:35,616 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:35,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/3d2519476d5a44b09dbd284b77c5910e 2023-04-07 09:55:35,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/00f283029e7c483da19045fd9f994dc8 2023-04-07 09:55:35,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/eac473439da544fba97fdab800d66518 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eac473439da544fba97fdab800d66518 2023-04-07 09:55:35,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eac473439da544fba97fdab800d66518, entries=250, sequenceid=43, filesize=16.2 K 2023-04-07 09:55:35,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/3d2519476d5a44b09dbd284b77c5910e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/3d2519476d5a44b09dbd284b77c5910e 2023-04-07 09:55:35,706 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/3d2519476d5a44b09dbd284b77c5910e, entries=150, sequenceid=43, filesize=11.6 K 2023-04-07 09:55:35,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/00f283029e7c483da19045fd9f994dc8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/00f283029e7c483da19045fd9f994dc8 2023-04-07 09:55:35,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/00f283029e7c483da19045fd9f994dc8, entries=150, sequenceid=43, filesize=11.6 K 2023-04-07 09:55:35,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c05568a99c8bbb1b0521e425f41a54f2 in 844ms, sequenceid=43, compaction requested=true 2023-04-07 09:55:35,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:35,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:35,710 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:35,710 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:35,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:35,711 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:35,711 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40334 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:35,711 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:35,711 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:35,711 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,712 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:35,712 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,712 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6d70dc3d333a4dc0896aee19a3920db1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc06d999145145a387d18a6de173f38e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eac473439da544fba97fdab800d66518] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=39.4 K 2023-04-07 09:55:35,712 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c777fbedb08e491dabe29339e97a0c51, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/deb109c6d6c5459e85fb03f9d8d39cf8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/3d2519476d5a44b09dbd284b77c5910e] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=32.5 K 2023-04-07 09:55:35,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 6d70dc3d333a4dc0896aee19a3920db1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680861334234 2023-04-07 09:55:35,712 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c777fbedb08e491dabe29339e97a0c51, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680861334237 2023-04-07 09:55:35,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bc06d999145145a387d18a6de173f38e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680861334243 2023-04-07 09:55:35,713 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting deb109c6d6c5459e85fb03f9d8d39cf8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680861334243 2023-04-07 09:55:35,713 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting eac473439da544fba97fdab800d66518, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1680861334861 2023-04-07 09:55:35,713 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3d2519476d5a44b09dbd284b77c5910e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1680861334861 2023-04-07 09:55:35,716 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:35,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:35,718 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:35,720 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:35,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:35,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:35,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:35,726 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:35,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:35,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:35,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,728 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:35,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:35,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:35,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:35,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:35,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:35,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:35,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:35,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:35,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:35,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:35,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:35,733 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,734 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:35,734 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 09:55:35,734 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:35,734 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:35,734 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:35,734 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:35,734 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:35,734 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:35,740 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/c73a501d7eac4a34bfc1eec0057397f6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c73a501d7eac4a34bfc1eec0057397f6 2023-04-07 09:55:35,745 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into c73a501d7eac4a34bfc1eec0057397f6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:35,745 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:35,745 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861335710; duration=0sec 2023-04-07 09:55:35,745 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:35,745 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:35,746 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:35,746 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:35,746 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:35,746 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/84447971fc6f40ff8e67c6f4d4d0cc97, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2b846a3bd91d4626828c552c72d05e81, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/00f283029e7c483da19045fd9f994dc8] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=32.5 K 2023-04-07 09:55:35,747 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 84447971fc6f40ff8e67c6f4d4d0cc97, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680861334237 2023-04-07 09:55:35,747 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 2b846a3bd91d4626828c552c72d05e81, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680861334243 2023-04-07 09:55:35,747 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 00f283029e7c483da19045fd9f994dc8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1680861334861 2023-04-07 09:55:35,752 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#461 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:36,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:36,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861396056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861396057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861396058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861396059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861396064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,141 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/8dc579032f1d483f8fed5dbb246373ef as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8dc579032f1d483f8fed5dbb246373ef 2023-04-07 09:55:36,144 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into 8dc579032f1d483f8fed5dbb246373ef(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:36,144 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:36,145 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861335710; duration=0sec 2023-04-07 09:55:36,145 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/04e98f42cdac422e86a72ea5eefad09e 2023-04-07 09:55:36,145 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:36,160 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/7d1b85930a3549058d5ffa606d6be7d2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7d1b85930a3549058d5ffa606d6be7d2 2023-04-07 09:55:36,163 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 7d1b85930a3549058d5ffa606d6be7d2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:36,164 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:36,164 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861335711; duration=0sec 2023-04-07 09:55:36,164 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861396165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861396165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861396165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861396169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861396172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861396372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861396372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861396373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861396374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861396381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,554 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/797351a67caa469b8407a58de29ab99c 2023-04-07 09:55:36,564 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e1171fc7e9894388963a8aec03a0fbfd 2023-04-07 09:55:36,567 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/04e98f42cdac422e86a72ea5eefad09e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/04e98f42cdac422e86a72ea5eefad09e 2023-04-07 09:55:36,570 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/04e98f42cdac422e86a72ea5eefad09e, entries=150, sequenceid=80, filesize=11.6 K 2023-04-07 09:55:36,571 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/797351a67caa469b8407a58de29ab99c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/797351a67caa469b8407a58de29ab99c 2023-04-07 09:55:36,574 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/797351a67caa469b8407a58de29ab99c, entries=150, sequenceid=80, filesize=11.6 K 2023-04-07 09:55:36,575 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e1171fc7e9894388963a8aec03a0fbfd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e1171fc7e9894388963a8aec03a0fbfd 2023-04-07 09:55:36,578 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e1171fc7e9894388963a8aec03a0fbfd, entries=150, sequenceid=80, filesize=11.6 K 2023-04-07 09:55:36,578 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for c05568a99c8bbb1b0521e425f41a54f2 in 844ms, sequenceid=80, compaction requested=false 2023-04-07 09:55:36,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:36,578 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:36,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:36,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:36,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:36,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:36,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24befe92[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:36,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:36,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,583 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:36,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:36,583 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:36,583 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,583 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:36,586 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:36,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:36,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:36,588 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:36,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:36,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:36,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:36,588 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:36,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:36,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:36,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:36,598 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:36,598 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,598 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:36,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:36,598 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:36,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:36,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:55:36,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:36,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:36,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:36,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:36,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:36,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:36,698 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:36,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:36,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:36,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:36,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:36,702 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:36,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:36,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:36,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:36,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,703 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:36,703 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:36,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:36,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:36,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:36,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:36,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:36,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:36,705 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:36,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:36,706 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:36,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,707 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,707 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,707 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:36,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69fb162c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:36,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:36,710 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,711 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:36,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861396702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,712 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:36,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861396703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:36,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861396709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:36,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:36,717 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:36,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:36,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:36,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:36,717 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:36,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861396710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,718 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861396711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,718 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,718 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:36,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:36,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:36,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:36,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:36,722 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,723 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:36,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,723 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:36,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:36,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861396813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861396813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861396815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861396819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:36,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861396819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,823 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:36,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:36,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:36,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:36,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:36,826 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:36,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:36,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:36,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:36,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,828 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:36,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,829 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:36,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:36,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:36,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:36,830 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:36,830 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:36,830 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:36,830 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:36,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:36,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:36,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77e1eb32[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:36,833 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:36,833 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:36,835 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:36,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:36,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:36,836 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:36,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:36,838 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:36,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:36,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:36,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:36,839 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:36,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:36,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:36,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:36,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,844 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:36,845 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:36,845 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:36,845 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,845 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,845 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:36,945 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:36,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:36,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:36,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:36,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:36,948 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:36,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:36,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:36,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:36,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,950 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,950 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:36,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:36,950 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,950 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:36,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:36,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:36,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:36,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:36,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:36,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:36,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:36,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:36,951 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:36,951 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:36,952 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:36,952 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:36,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:36,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:36,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:36,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:36,953 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:36,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f8e96e4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:36,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:36,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:36,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:36,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:36,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:36,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:36,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:36,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:36,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:36,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:36,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:36,962 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,962 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,962 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:36,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:36,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:36,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:36,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:36,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:36,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:36,967 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:36,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:36,967 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:36,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:36,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861397019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861397019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861397019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861397023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861397024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,067 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:37,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:37,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:37,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:37,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:37,083 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:37,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:37,084 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:37,084 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,084 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/33ee14bd92354848bb6f872e8772f71f 2023-04-07 09:55:37,089 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,089 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:37,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:37,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:37,091 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,091 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:37,091 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:37,091 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:37,091 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:37,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c17bbe9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:37,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:37,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,097 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:37,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:37,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:37,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,098 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:37,100 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,100 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:37,100 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,100 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:37,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:37,102 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:37,102 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:37,102 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:37,102 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:37,102 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:37,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/908d20ba5a804a98a1bc06229d0a6e02 2023-04-07 09:55:37,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:37,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:37,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:37,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,111 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,111 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,111 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:37,111 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:37,111 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,111 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,211 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:37,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:37,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:37,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:37,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:37,214 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:37,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:37,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:37,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,214 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,215 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,215 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:37,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:37,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:37,217 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,217 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:37,217 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:37,217 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:37,218 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 09:55:37,219 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:37,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:37,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d6c9431[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:37,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:37,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,223 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:37,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:37,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:37,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:37,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:37,230 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:37,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:37,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:37,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:37,231 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:37,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,234 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,239 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:37,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:37,239 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:37,239 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:37,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:37,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861397322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861397322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861397322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861397327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861397327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,339 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:37,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:37,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:37,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:37,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:37,342 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:37,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:37,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:37,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,344 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:37,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:37,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:37,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:37,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,347 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@552c42d5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:37,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:37,349 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,350 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:37,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:37,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:37,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:37,355 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:37,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:37,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:37,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:37,355 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:37,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,357 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,360 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:37,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:37,360 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:37,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:37,360 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:37,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,460 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:37,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:37,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:37,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:37,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:37,463 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:37,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:37,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:37,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,465 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,465 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:37,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:37,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:37,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:37,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:37,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:37,467 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,467 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:37,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:37,467 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:37,467 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:37,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:37,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,474 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,475 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:37,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:37,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2ec722[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:37,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:37,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,480 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:37,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:37,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:37,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:37,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:37,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:37,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:37,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:37,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:37,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:37,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:37,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,492 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:37,492 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:37,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:37,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/d099d6c348864f5b967e664d17bf7131 2023-04-07 09:55:37,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/33ee14bd92354848bb6f872e8772f71f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/33ee14bd92354848bb6f872e8772f71f 2023-04-07 09:55:37,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/33ee14bd92354848bb6f872e8772f71f, entries=200, sequenceid=95, filesize=13.9 K 2023-04-07 09:55:37,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/908d20ba5a804a98a1bc06229d0a6e02 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/908d20ba5a804a98a1bc06229d0a6e02 2023-04-07 09:55:37,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/908d20ba5a804a98a1bc06229d0a6e02, entries=150, sequenceid=95, filesize=11.6 K 2023-04-07 09:55:37,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/d099d6c348864f5b967e664d17bf7131 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d099d6c348864f5b967e664d17bf7131 2023-04-07 09:55:37,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d099d6c348864f5b967e664d17bf7131, entries=150, sequenceid=95, filesize=11.6 K 2023-04-07 09:55:37,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c05568a99c8bbb1b0521e425f41a54f2 in 857ms, sequenceid=95, compaction requested=true 2023-04-07 09:55:37,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:37,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:37,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:37,536 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:37,536 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:37,536 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:37,537 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:37,537 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:37,537 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:37,537 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:37,537 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,537 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,537 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c73a501d7eac4a34bfc1eec0057397f6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/04e98f42cdac422e86a72ea5eefad09e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/33ee14bd92354848bb6f872e8772f71f] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=37.2 K 2023-04-07 09:55:37,537 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8dc579032f1d483f8fed5dbb246373ef, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/797351a67caa469b8407a58de29ab99c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/908d20ba5a804a98a1bc06229d0a6e02] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=34.9 K 2023-04-07 09:55:37,537 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting c73a501d7eac4a34bfc1eec0057397f6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1680861334861 2023-04-07 09:55:37,537 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8dc579032f1d483f8fed5dbb246373ef, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1680861334861 2023-04-07 09:55:37,538 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 04e98f42cdac422e86a72ea5eefad09e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680861334919 2023-04-07 09:55:37,538 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 797351a67caa469b8407a58de29ab99c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680861334919 2023-04-07 09:55:37,538 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 33ee14bd92354848bb6f872e8772f71f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680861336056 2023-04-07 09:55:37,538 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 908d20ba5a804a98a1bc06229d0a6e02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680861336056 2023-04-07 09:55:37,542 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#468 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:37,543 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:37,549 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/7b5456047ef5452398bef3c46d0e9811 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/7b5456047ef5452398bef3c46d0e9811 2023-04-07 09:55:37,552 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into 7b5456047ef5452398bef3c46d0e9811(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:37,552 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:37,553 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861337536; duration=0sec 2023-04-07 09:55:37,553 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:37,553 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:37,553 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:37,553 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:37,554 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,554 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7d1b85930a3549058d5ffa606d6be7d2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e1171fc7e9894388963a8aec03a0fbfd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d099d6c348864f5b967e664d17bf7131] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=34.9 K 2023-04-07 09:55:37,554 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7d1b85930a3549058d5ffa606d6be7d2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1680861334861 2023-04-07 09:55:37,554 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e1171fc7e9894388963a8aec03a0fbfd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680861334919 2023-04-07 09:55:37,554 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d099d6c348864f5b967e664d17bf7131, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680861336056 2023-04-07 09:55:37,558 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#469 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:37,592 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:37,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:37,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:37,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:37,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:37,596 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:37,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:37,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:37,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,599 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:37,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:37,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:37,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:37,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:37,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:37,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:37,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:37,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:37,601 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:55:37,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:37,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:37,601 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:37,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:37,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:37,602 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:37,609 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/d99e69e47f65492681c3755b65b5de26 2023-04-07 09:55:37,621 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/b150ff0cf4ee47ceac2ba0f7fe909318 2023-04-07 09:55:37,633 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/883602856e334d2b9c6c92c587723f33 2023-04-07 09:55:37,638 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/d99e69e47f65492681c3755b65b5de26 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d99e69e47f65492681c3755b65b5de26 2023-04-07 09:55:37,641 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d99e69e47f65492681c3755b65b5de26, entries=150, sequenceid=116, filesize=11.6 K 2023-04-07 09:55:37,641 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/b150ff0cf4ee47ceac2ba0f7fe909318 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/b150ff0cf4ee47ceac2ba0f7fe909318 2023-04-07 09:55:37,645 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/b150ff0cf4ee47ceac2ba0f7fe909318, entries=150, sequenceid=116, filesize=11.6 K 2023-04-07 09:55:37,645 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/883602856e334d2b9c6c92c587723f33 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/883602856e334d2b9c6c92c587723f33 2023-04-07 09:55:37,649 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/883602856e334d2b9c6c92c587723f33, entries=150, sequenceid=116, filesize=11.6 K 2023-04-07 09:55:37,650 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for c05568a99c8bbb1b0521e425f41a54f2 in 49ms, sequenceid=116, compaction requested=false 2023-04-07 09:55:37,650 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:37,650 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:37,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:37,651 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,652 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,652 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,652 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:37,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc7f1fd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:37,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:37,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:37,655 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:37,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,655 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:37,655 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:37,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:37,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:37,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:37,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:37,658 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:37,659 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:37,659 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:37,659 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:37,659 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:37,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,662 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:37,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:37,665 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:37,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,665 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,666 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,766 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:37,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:37,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:37,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:37,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:37,769 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:37,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:37,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:37,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,770 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:37,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:37,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:37,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:37,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:37,772 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:37,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:37,773 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,774 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:37,775 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:37,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c32928[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:37,775 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,777 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:37,778 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:37,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:37,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,779 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,779 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,779 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:37,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:37,781 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:37,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:37,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:37,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:37,781 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:37,782 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:37,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:37,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:37,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:37,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,787 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:37,787 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:37,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:37,839 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:37,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:37,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:37,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:37,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:37,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:37,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:37,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861337857, max timestamp=1680861337866Num uniques 859; 2023-04-07 09:55:37,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:55:37,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861337857, max timestamp=1680861337866Num uniques 859; 2023-04-07 09:55:37,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:55:37,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680861337857, max timestamp=1680861337866Num uniques 859; 2023-04-07 09:55:37,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-07 09:55:37,887 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:37,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:37,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:37,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:37,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:37,890 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:37,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:37,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:37,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,891 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861397881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,893 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:37,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:37,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:37,894 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:37,894 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:37,894 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:37,896 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,896 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:37,896 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:37,896 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:37,896 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:37,898 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,898 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,898 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:37,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861397889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,900 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861397889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861397890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861397892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,900 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:37,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@672651ca[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:37,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:37,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:37,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,902 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:37,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:37,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:37,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:37,907 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:37,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:37,909 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:37,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:37,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:37,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:37,910 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:37,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:37,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:37,913 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:37,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:37,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:37,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:37,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:37,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:37,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:37,916 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:37,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:37,917 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:37,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:37,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:37,951 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/1825532a92654c16864d44a9554018f3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/1825532a92654c16864d44a9554018f3 2023-04-07 09:55:37,954 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 1825532a92654c16864d44a9554018f3(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:37,954 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:37,954 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861337536; duration=0sec 2023-04-07 09:55:37,954 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:37,969 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/1f0fb050e27a4b3f8d97ff0de511370e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1f0fb050e27a4b3f8d97ff0de511370e 2023-04-07 09:55:37,972 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 1f0fb050e27a4b3f8d97ff0de511370e(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:37,972 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:37,972 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861337536; duration=0sec 2023-04-07 09:55:37,972 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:37,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:37,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861397993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861398001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861398001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861398001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861398002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,017 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:38,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:38,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:38,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:38,020 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:38,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:38,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:38,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,020 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,021 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:38,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:38,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:38,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:38,023 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:38,023 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:38,023 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:38,023 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:38,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:38,026 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,026 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,026 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,026 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:38,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:38,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:38,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31cee9cf[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:38,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,028 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,028 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:38,028 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:38,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:38,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:38,030 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:38,030 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,030 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,030 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:38,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:38,032 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:38,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:38,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:38,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:38,032 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:38,033 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,033 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,033 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,034 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:38,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:38,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,038 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:38,038 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:38,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:38,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,138 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:38,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:38,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:38,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:38,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:38,141 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:38,141 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,141 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:38,142 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:38,142 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,142 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,143 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,143 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:38,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:38,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:38,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:38,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:38,144 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:38,144 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:38,145 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,145 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:38,145 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:38,145 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:38,145 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:38,146 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,146 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,147 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,147 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:38,148 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c13f03[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:38,148 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,149 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,149 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:38,149 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:38,149 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:38,149 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,149 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:38,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:38,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:38,153 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:38,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:38,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:38,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:38,154 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:38,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,155 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,157 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,159 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:38,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,159 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:38,159 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:38,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861398200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861398208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861398208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861398209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861398209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/aebae9ffd1974802a5a5d6a05e227db1 2023-04-07 09:55:38,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/9a6ecb6c15534471bf68af76b7475b2b 2023-04-07 09:55:38,260 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:38,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:38,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/350f6d1957e84df88adab89619b6dd43 2023-04-07 09:55:38,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:38,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:38,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:38,264 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:38,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:38,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:38,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/aebae9ffd1974802a5a5d6a05e227db1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/aebae9ffd1974802a5a5d6a05e227db1 2023-04-07 09:55:38,265 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,265 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,266 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,266 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:38,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:38,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:38,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:38,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:38,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:38,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:38,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:38,267 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,267 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:38,268 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:38,268 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:38,268 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:38,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/aebae9ffd1974802a5a5d6a05e227db1, entries=200, sequenceid=127, filesize=13.9 K 2023-04-07 09:55:38,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/9a6ecb6c15534471bf68af76b7475b2b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9a6ecb6c15534471bf68af76b7475b2b 2023-04-07 09:55:38,270 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,270 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:38,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f97bdf[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:38,272 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:38,272 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9a6ecb6c15534471bf68af76b7475b2b, entries=150, sequenceid=127, filesize=11.6 K 2023-04-07 09:55:38,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:38,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:38,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,273 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:38,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:38,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/350f6d1957e84df88adab89619b6dd43 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/350f6d1957e84df88adab89619b6dd43 2023-04-07 09:55:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,274 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,274 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/350f6d1957e84df88adab89619b6dd43, entries=150, sequenceid=127, filesize=11.6 K 2023-04-07 09:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c05568a99c8bbb1b0521e425f41a54f2 in 438ms, sequenceid=127, compaction requested=true 2023-04-07 09:55:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:38,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:38,277 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:38,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:38,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:38,277 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:38,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:38,277 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:38,277 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:38,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:38,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:38,278 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:38,279 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:38,279 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:38,279 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:38,279 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,279 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:38,279 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/1825532a92654c16864d44a9554018f3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d99e69e47f65492681c3755b65b5de26, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/aebae9ffd1974802a5a5d6a05e227db1] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=37.3 K 2023-04-07 09:55:38,279 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,279 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/7b5456047ef5452398bef3c46d0e9811, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/b150ff0cf4ee47ceac2ba0f7fe909318, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9a6ecb6c15534471bf68af76b7475b2b] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=35.0 K 2023-04-07 09:55:38,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,279 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1825532a92654c16864d44a9554018f3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680861336056 2023-04-07 09:55:38,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,280 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d99e69e47f65492681c3755b65b5de26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680861336700 2023-04-07 09:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,280 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7b5456047ef5452398bef3c46d0e9811, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680861336056 2023-04-07 09:55:38,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,280 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting aebae9ffd1974802a5a5d6a05e227db1, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680861337830 2023-04-07 09:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,280 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b150ff0cf4ee47ceac2ba0f7fe909318, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680861336700 2023-04-07 09:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,281 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9a6ecb6c15534471bf68af76b7475b2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680861337830 2023-04-07 09:55:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,281 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,287 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:38,287 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:38,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,289 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,289 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:38,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:38,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,289 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:38,390 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:38,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:38,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:38,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:38,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:38,393 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:38,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:38,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:38,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,393 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,394 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:38,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:38,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:38,396 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,396 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:38,396 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-07 09:55:38,396 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:38,396 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:38,396 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:38,397 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:38,397 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:38,397 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:38,403 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/eaf9a1169d964717a1f15e75bcbd2e39 2023-04-07 09:55:38,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:38,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:38,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861398524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861398525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861398526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861398526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861398533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861398634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861398634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861398637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861398638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861398638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,703 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/3df295549f7c43d49861a1236e12df4e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3df295549f7c43d49861a1236e12df4e 2023-04-07 09:55:38,704 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/9c186eee900348a68569edd9cf53e149 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9c186eee900348a68569edd9cf53e149 2023-04-07 09:55:38,708 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 3df295549f7c43d49861a1236e12df4e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:38,708 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:38,708 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861338277; duration=0sec 2023-04-07 09:55:38,709 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:38,709 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:38,709 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into 9c186eee900348a68569edd9cf53e149(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:38,709 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:38,709 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861338277; duration=0sec 2023-04-07 09:55:38,709 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:38,710 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:38,711 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:38,711 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,711 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1f0fb050e27a4b3f8d97ff0de511370e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/883602856e334d2b9c6c92c587723f33, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/350f6d1957e84df88adab89619b6dd43] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=35.0 K 2023-04-07 09:55:38,711 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1f0fb050e27a4b3f8d97ff0de511370e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680861336056 2023-04-07 09:55:38,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 883602856e334d2b9c6c92c587723f33, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680861336700 2023-04-07 09:55:38,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 350f6d1957e84df88adab89619b6dd43, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680861337830 2023-04-07 09:55:38,716 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#480 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:38,812 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/e442bc974854446e99ac3b4f7759200b 2023-04-07 09:55:38,820 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/90250a17dae64de9851e2166ee3ae750 2023-04-07 09:55:38,823 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/eaf9a1169d964717a1f15e75bcbd2e39 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eaf9a1169d964717a1f15e75bcbd2e39 2023-04-07 09:55:38,828 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eaf9a1169d964717a1f15e75bcbd2e39, entries=150, sequenceid=164, filesize=11.8 K 2023-04-07 09:55:38,829 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/e442bc974854446e99ac3b4f7759200b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e442bc974854446e99ac3b4f7759200b 2023-04-07 09:55:38,833 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e442bc974854446e99ac3b4f7759200b, entries=150, sequenceid=164, filesize=11.8 K 2023-04-07 09:55:38,833 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/90250a17dae64de9851e2166ee3ae750 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/90250a17dae64de9851e2166ee3ae750 2023-04-07 09:55:38,836 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/90250a17dae64de9851e2166ee3ae750, entries=150, sequenceid=164, filesize=11.8 K 2023-04-07 09:55:38,837 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for c05568a99c8bbb1b0521e425f41a54f2 in 441ms, sequenceid=164, compaction requested=false 2023-04-07 09:55:38,837 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:38,837 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:38,839 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:38,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ec83a1[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:38,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:38,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,843 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:38,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:38,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:55:38,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:38,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:38,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:38,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:38,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:38,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,846 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:38,850 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:38,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:38,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:38,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:38,850 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:38,851 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,851 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,851 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/61988be14c91418d9c661941eb0271e6 2023-04-07 09:55:38,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,856 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:38,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:38,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:38,856 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,857 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:38,857 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861398867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861398868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861398871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861398872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861398874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,957 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:38,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:38,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:38,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:38,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:38,960 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:38,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:38,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:38,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,961 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:38,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:38,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:38,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:38,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:38,963 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:38,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:38,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:38,970 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:38,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa9a22[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:38,970 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,971 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,971 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:38,972 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:38,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:38,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:38,973 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:38,973 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,973 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:38,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:38,976 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:38,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:38,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:38,976 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:38,976 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:38,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:38,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:38,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:38,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:38,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861398975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:38,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861398975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861398975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:38,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:38,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:38,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:38,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:38,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:38,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861398979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:38,985 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:38,985 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:38,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:38,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:38,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861398979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:38,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:38,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,085 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:39,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:39,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:39,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:39,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:39,089 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:39,089 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,089 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:39,089 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:39,089 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,089 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,090 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:39,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:39,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:39,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:39,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:39,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:39,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:39,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:39,092 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:39,092 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,092 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:39,092 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:39,092 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:39,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:39,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:39,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:39,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,094 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:39,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20339413[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:39,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:39,096 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,097 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:39,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:39,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:39,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,097 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:39,099 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:39,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,100 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:39,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:39,103 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:39,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:39,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:39,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:39,103 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:39,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,112 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:39,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:39,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,113 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:39,113 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:39,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:39,124 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/d4a1ad6bc2f548adbdfe3a79dd92592e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d4a1ad6bc2f548adbdfe3a79dd92592e 2023-04-07 09:55:39,127 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into d4a1ad6bc2f548adbdfe3a79dd92592e(size=11.9 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:39,128 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:39,128 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861338278; duration=0sec 2023-04-07 09:55:39,128 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:39,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861399181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861399183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861399183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861399186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861399186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,213 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:39,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:39,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:39,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:39,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:39,216 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:39,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:39,216 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:39,217 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,217 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,219 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:39,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:39,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:39,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:39,220 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:39,221 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,221 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:39,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:39,221 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:39,221 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:39,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:39,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:39,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,224 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:39,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:39,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32063449[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:39,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,229 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:39,231 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:39,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:39,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:39,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:39,231 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:39,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,237 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,238 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,239 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:39,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,239 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:39,239 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/5146a5fd4ad444c688983f7a09668091 2023-04-07 09:55:39,339 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:39,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:39,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:39,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:39,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:39,342 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:39,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:39,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:39,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,344 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:39,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:39,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:39,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:39,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:39,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:39,346 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:39,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:39,348 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,348 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,349 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:39,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:39,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d54823[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:39,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,353 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,353 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,353 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:39,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:39,354 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:39,354 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:39,355 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:39,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:39,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:39,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:39,358 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:39,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:39,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:39,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:39,358 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:39,359 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,360 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,364 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:39,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,364 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,364 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:39,364 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:39,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,464 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:39,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:39,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:39,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:39,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:39,467 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:39,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:39,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:39,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,471 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:39,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:39,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:39,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:39,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:39,472 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:39,472 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:39,473 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:39,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@447abb91[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:39,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:39,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:39,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:39,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:39,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:39,477 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,477 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:39,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:39,479 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:39,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:39,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:39,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:39,480 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:39,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,483 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:39,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:39,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,486 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,487 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:39,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,487 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:39,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861399489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861399490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861399490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861399490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861399492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,587 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:39,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:39,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:39,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:39,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:39,591 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:39,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:39,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:39,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,595 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,596 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:39,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:39,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:39,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:39,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:39,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:39,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:39,600 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,600 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:39,600 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:39,600 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:39,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:39,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:39,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:39,607 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:39,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78553c01[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:39,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:39,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:39,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:39,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:39,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:39,611 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,611 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:39,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:39,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:39,615 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:39,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:39,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:39,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:39,615 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:39,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:39,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:39,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,620 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:39,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:39,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:39,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:39,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,624 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:39,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:39,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:39,624 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:39,624 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/12296281a2a84a269e4006b1b8472f60 2023-04-07 09:55:39,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/61988be14c91418d9c661941eb0271e6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/61988be14c91418d9c661941eb0271e6 2023-04-07 09:55:39,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/61988be14c91418d9c661941eb0271e6, entries=300, sequenceid=179, filesize=18.8 K 2023-04-07 09:55:39,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/5146a5fd4ad444c688983f7a09668091 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/5146a5fd4ad444c688983f7a09668091 2023-04-07 09:55:39,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/5146a5fd4ad444c688983f7a09668091, entries=150, sequenceid=179, filesize=11.8 K 2023-04-07 09:55:39,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/12296281a2a84a269e4006b1b8472f60 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/12296281a2a84a269e4006b1b8472f60 2023-04-07 09:55:39,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/12296281a2a84a269e4006b1b8472f60, entries=150, sequenceid=179, filesize=11.8 K 2023-04-07 09:55:39,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c05568a99c8bbb1b0521e425f41a54f2 in 851ms, sequenceid=179, compaction requested=true 2023-04-07 09:55:39,695 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:39,695 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:39,695 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:39,695 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:39,695 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:39,695 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:39,696 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43436 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:39,696 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:39,696 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,696 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3df295549f7c43d49861a1236e12df4e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eaf9a1169d964717a1f15e75bcbd2e39, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/61988be14c91418d9c661941eb0271e6] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=42.4 K 2023-04-07 09:55:39,697 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:39,697 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:39,697 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3df295549f7c43d49861a1236e12df4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680861337830 2023-04-07 09:55:39,697 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,697 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9c186eee900348a68569edd9cf53e149, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e442bc974854446e99ac3b4f7759200b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/5146a5fd4ad444c688983f7a09668091] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=35.4 K 2023-04-07 09:55:39,697 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting eaf9a1169d964717a1f15e75bcbd2e39, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1680861337876 2023-04-07 09:55:39,697 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 61988be14c91418d9c661941eb0271e6, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680861338524 2023-04-07 09:55:39,697 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9c186eee900348a68569edd9cf53e149, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680861337830 2023-04-07 09:55:39,698 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e442bc974854446e99ac3b4f7759200b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1680861337876 2023-04-07 09:55:39,699 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 5146a5fd4ad444c688983f7a09668091, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680861338524 2023-04-07 09:55:39,705 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#485 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:39,721 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:39,724 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:39,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:39,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:39,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:39,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:39,729 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:39,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:39,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:39,730 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,730 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,731 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/81863bf166544ae0b40a0dbc6000cb52 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/81863bf166544ae0b40a0dbc6000cb52 2023-04-07 09:55:39,731 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,732 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:39,732 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:39,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:39,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:39,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:39,736 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into 81863bf166544ae0b40a0dbc6000cb52(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:39,736 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:39,736 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861339695; duration=0sec 2023-04-07 09:55:39,736 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:39,736 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:39,738 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:39,738 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:39,738 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,738 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d4a1ad6bc2f548adbdfe3a79dd92592e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/90250a17dae64de9851e2166ee3ae750, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/12296281a2a84a269e4006b1b8472f60] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=35.4 K 2023-04-07 09:55:39,739 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d4a1ad6bc2f548adbdfe3a79dd92592e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680861337830 2023-04-07 09:55:39,739 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 90250a17dae64de9851e2166ee3ae750, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1680861337876 2023-04-07 09:55:39,739 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 12296281a2a84a269e4006b1b8472f60, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680861338524 2023-04-07 09:55:39,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:39,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:39,748 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:39,750 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#487 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:39,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:39,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:39,751 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:39,751 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:39,751 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:39,752 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:39,771 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/81bfff3eb42d4a87bbe3c947947c619d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81bfff3eb42d4a87bbe3c947947c619d 2023-04-07 09:55:39,779 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 81bfff3eb42d4a87bbe3c947947c619d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:39,779 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:39,779 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861339695; duration=0sec 2023-04-07 09:55:39,779 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:40,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:40,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:40,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861400016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861400017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861400017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861400018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861400025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,121 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/e3b319390c7846e6aab03df04f570e0c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/e3b319390c7846e6aab03df04f570e0c 2023-04-07 09:55:40,126 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into e3b319390c7846e6aab03df04f570e0c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:40,127 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:40,127 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861339695; duration=0sec 2023-04-07 09:55:40,127 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:40,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861400126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861400129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861400129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861400129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861400132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,179 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/a5ad922143e249f484608682651a8500 2023-04-07 09:55:40,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861400333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861400336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861400337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861400337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861400340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,449 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-07 09:55:40,474 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6d70dc3d333a4dc0896aee19a3920db1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc06d999145145a387d18a6de173f38e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eac473439da544fba97fdab800d66518, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c73a501d7eac4a34bfc1eec0057397f6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/04e98f42cdac422e86a72ea5eefad09e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/33ee14bd92354848bb6f872e8772f71f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/1825532a92654c16864d44a9554018f3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d99e69e47f65492681c3755b65b5de26, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/aebae9ffd1974802a5a5d6a05e227db1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3df295549f7c43d49861a1236e12df4e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eaf9a1169d964717a1f15e75bcbd2e39, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/61988be14c91418d9c661941eb0271e6] to archive 2023-04-07 09:55:40,475 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:40,477 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6d70dc3d333a4dc0896aee19a3920db1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6d70dc3d333a4dc0896aee19a3920db1 2023-04-07 09:55:40,478 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc06d999145145a387d18a6de173f38e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc06d999145145a387d18a6de173f38e 2023-04-07 09:55:40,480 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eac473439da544fba97fdab800d66518 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eac473439da544fba97fdab800d66518 2023-04-07 09:55:40,481 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c73a501d7eac4a34bfc1eec0057397f6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c73a501d7eac4a34bfc1eec0057397f6 2023-04-07 09:55:40,483 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/04e98f42cdac422e86a72ea5eefad09e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/04e98f42cdac422e86a72ea5eefad09e 2023-04-07 09:55:40,484 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/33ee14bd92354848bb6f872e8772f71f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/33ee14bd92354848bb6f872e8772f71f 2023-04-07 09:55:40,485 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/1825532a92654c16864d44a9554018f3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/1825532a92654c16864d44a9554018f3 2023-04-07 09:55:40,486 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d99e69e47f65492681c3755b65b5de26 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d99e69e47f65492681c3755b65b5de26 2023-04-07 09:55:40,487 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/aebae9ffd1974802a5a5d6a05e227db1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/aebae9ffd1974802a5a5d6a05e227db1 2023-04-07 09:55:40,488 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3df295549f7c43d49861a1236e12df4e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3df295549f7c43d49861a1236e12df4e 2023-04-07 09:55:40,489 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eaf9a1169d964717a1f15e75bcbd2e39 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/eaf9a1169d964717a1f15e75bcbd2e39 2023-04-07 09:55:40,492 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/61988be14c91418d9c661941eb0271e6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/61988be14c91418d9c661941eb0271e6 2023-04-07 09:55:40,516 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c777fbedb08e491dabe29339e97a0c51, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/deb109c6d6c5459e85fb03f9d8d39cf8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8dc579032f1d483f8fed5dbb246373ef, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/3d2519476d5a44b09dbd284b77c5910e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/797351a67caa469b8407a58de29ab99c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/7b5456047ef5452398bef3c46d0e9811, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/908d20ba5a804a98a1bc06229d0a6e02, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/b150ff0cf4ee47ceac2ba0f7fe909318, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9c186eee900348a68569edd9cf53e149, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9a6ecb6c15534471bf68af76b7475b2b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e442bc974854446e99ac3b4f7759200b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/5146a5fd4ad444c688983f7a09668091] to archive 2023-04-07 09:55:40,517 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:40,520 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c777fbedb08e491dabe29339e97a0c51 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c777fbedb08e491dabe29339e97a0c51 2023-04-07 09:55:40,521 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/deb109c6d6c5459e85fb03f9d8d39cf8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/deb109c6d6c5459e85fb03f9d8d39cf8 2023-04-07 09:55:40,522 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8dc579032f1d483f8fed5dbb246373ef to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8dc579032f1d483f8fed5dbb246373ef 2023-04-07 09:55:40,524 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/3d2519476d5a44b09dbd284b77c5910e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/3d2519476d5a44b09dbd284b77c5910e 2023-04-07 09:55:40,525 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/797351a67caa469b8407a58de29ab99c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/797351a67caa469b8407a58de29ab99c 2023-04-07 09:55:40,526 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/7b5456047ef5452398bef3c46d0e9811 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/7b5456047ef5452398bef3c46d0e9811 2023-04-07 09:55:40,527 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/908d20ba5a804a98a1bc06229d0a6e02 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/908d20ba5a804a98a1bc06229d0a6e02 2023-04-07 09:55:40,529 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/b150ff0cf4ee47ceac2ba0f7fe909318 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/b150ff0cf4ee47ceac2ba0f7fe909318 2023-04-07 09:55:40,530 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9c186eee900348a68569edd9cf53e149 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9c186eee900348a68569edd9cf53e149 2023-04-07 09:55:40,531 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9a6ecb6c15534471bf68af76b7475b2b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9a6ecb6c15534471bf68af76b7475b2b 2023-04-07 09:55:40,532 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e442bc974854446e99ac3b4f7759200b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e442bc974854446e99ac3b4f7759200b 2023-04-07 09:55:40,535 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/5146a5fd4ad444c688983f7a09668091 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/5146a5fd4ad444c688983f7a09668091 2023-04-07 09:55:40,538 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/84447971fc6f40ff8e67c6f4d4d0cc97, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2b846a3bd91d4626828c552c72d05e81, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7d1b85930a3549058d5ffa606d6be7d2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/00f283029e7c483da19045fd9f994dc8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e1171fc7e9894388963a8aec03a0fbfd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1f0fb050e27a4b3f8d97ff0de511370e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d099d6c348864f5b967e664d17bf7131, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/883602856e334d2b9c6c92c587723f33, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d4a1ad6bc2f548adbdfe3a79dd92592e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/350f6d1957e84df88adab89619b6dd43, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/90250a17dae64de9851e2166ee3ae750, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/12296281a2a84a269e4006b1b8472f60] to archive 2023-04-07 09:55:40,539 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:40,540 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/84447971fc6f40ff8e67c6f4d4d0cc97 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/84447971fc6f40ff8e67c6f4d4d0cc97 2023-04-07 09:55:40,541 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2b846a3bd91d4626828c552c72d05e81 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2b846a3bd91d4626828c552c72d05e81 2023-04-07 09:55:40,542 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7d1b85930a3549058d5ffa606d6be7d2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7d1b85930a3549058d5ffa606d6be7d2 2023-04-07 09:55:40,544 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/00f283029e7c483da19045fd9f994dc8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/00f283029e7c483da19045fd9f994dc8 2023-04-07 09:55:40,545 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e1171fc7e9894388963a8aec03a0fbfd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e1171fc7e9894388963a8aec03a0fbfd 2023-04-07 09:55:40,546 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1f0fb050e27a4b3f8d97ff0de511370e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1f0fb050e27a4b3f8d97ff0de511370e 2023-04-07 09:55:40,547 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d099d6c348864f5b967e664d17bf7131 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d099d6c348864f5b967e664d17bf7131 2023-04-07 09:55:40,550 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/883602856e334d2b9c6c92c587723f33 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/883602856e334d2b9c6c92c587723f33 2023-04-07 09:55:40,551 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d4a1ad6bc2f548adbdfe3a79dd92592e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d4a1ad6bc2f548adbdfe3a79dd92592e 2023-04-07 09:55:40,552 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/350f6d1957e84df88adab89619b6dd43 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/350f6d1957e84df88adab89619b6dd43 2023-04-07 09:55:40,553 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/90250a17dae64de9851e2166ee3ae750 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/90250a17dae64de9851e2166ee3ae750 2023-04-07 09:55:40,554 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/12296281a2a84a269e4006b1b8472f60 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/12296281a2a84a269e4006b1b8472f60 2023-04-07 09:55:40,591 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/0d8637e298b04ce1b5016d2180651f61 2023-04-07 09:55:40,610 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/8764bd7e6cea48a4adf6c0520921adaf 2023-04-07 09:55:40,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/a5ad922143e249f484608682651a8500 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/a5ad922143e249f484608682651a8500 2023-04-07 09:55:40,626 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/a5ad922143e249f484608682651a8500, entries=150, sequenceid=200, filesize=11.8 K 2023-04-07 09:55:40,627 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/0d8637e298b04ce1b5016d2180651f61 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0d8637e298b04ce1b5016d2180651f61 2023-04-07 09:55:40,631 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0d8637e298b04ce1b5016d2180651f61, entries=150, sequenceid=200, filesize=11.8 K 2023-04-07 09:55:40,632 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/8764bd7e6cea48a4adf6c0520921adaf as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8764bd7e6cea48a4adf6c0520921adaf 2023-04-07 09:55:40,638 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8764bd7e6cea48a4adf6c0520921adaf, entries=150, sequenceid=200, filesize=11.8 K 2023-04-07 09:55:40,638 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for c05568a99c8bbb1b0521e425f41a54f2 in 887ms, sequenceid=200, compaction requested=false 2023-04-07 09:55:40,638 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:40,638 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:40,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:40,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf2eb3e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:40,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:40,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:40,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:40,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:40,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:40,645 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-07 09:55:40,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:40,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:40,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:40,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:40,646 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:40,646 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:40,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:40,648 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:40,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:40,652 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:40,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:40,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:40,652 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:40,652 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:40,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:40,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,654 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,655 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,660 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:40,660 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:40,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:40,660 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,660 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:40,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:40,660 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:40,660 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:40,660 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:40,660 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:40,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:40,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/bc11397083a44bafb8188fb3f8a9b68a 2023-04-07 09:55:40,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861400664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861400668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861400672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861400674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861400675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/157230b72aa34964bb793ace3dc808ee 2023-04-07 09:55:40,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e8f0d6af560f442193902223365dcbde 2023-04-07 09:55:40,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/bc11397083a44bafb8188fb3f8a9b68a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc11397083a44bafb8188fb3f8a9b68a 2023-04-07 09:55:40,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc11397083a44bafb8188fb3f8a9b68a, entries=200, sequenceid=217, filesize=14.1 K 2023-04-07 09:55:40,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/157230b72aa34964bb793ace3dc808ee as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/157230b72aa34964bb793ace3dc808ee 2023-04-07 09:55:40,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/157230b72aa34964bb793ace3dc808ee, entries=150, sequenceid=217, filesize=11.8 K 2023-04-07 09:55:40,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e8f0d6af560f442193902223365dcbde as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8f0d6af560f442193902223365dcbde 2023-04-07 09:55:40,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8f0d6af560f442193902223365dcbde, entries=150, sequenceid=217, filesize=11.8 K 2023-04-07 09:55:40,751 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c05568a99c8bbb1b0521e425f41a54f2 in 106ms, sequenceid=217, compaction requested=true 2023-04-07 09:55:40,751 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:40,751 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:40,751 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:40,751 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:40,751 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:40,751 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:40,752 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:40,752 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:40,753 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:40,753 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:40,753 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,753 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,753 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/e3b319390c7846e6aab03df04f570e0c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/a5ad922143e249f484608682651a8500, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc11397083a44bafb8188fb3f8a9b68a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=38.0 K 2023-04-07 09:55:40,753 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/81863bf166544ae0b40a0dbc6000cb52, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0d8637e298b04ce1b5016d2180651f61, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/157230b72aa34964bb793ace3dc808ee] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=35.7 K 2023-04-07 09:55:40,753 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e3b319390c7846e6aab03df04f570e0c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680861338524 2023-04-07 09:55:40,753 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 81863bf166544ae0b40a0dbc6000cb52, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680861338524 2023-04-07 09:55:40,754 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting a5ad922143e249f484608682651a8500, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680861338870 2023-04-07 09:55:40,754 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0d8637e298b04ce1b5016d2180651f61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680861338870 2023-04-07 09:55:40,754 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bc11397083a44bafb8188fb3f8a9b68a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680861340006 2023-04-07 09:55:40,754 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 157230b72aa34964bb793ace3dc808ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680861340006 2023-04-07 09:55:40,761 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:40,761 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:40,768 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:40,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:40,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:40,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:40,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:40,776 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:40,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:40,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:40,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:40,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:40,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:40,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:40,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:40,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:40,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 09:55:40,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:40,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:40,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:40,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:40,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:40,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:40,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:40,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:40,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:40,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:40,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:40,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:40,820 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,820 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:40,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:40,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:40,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:40,828 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/237ccaa45bca4b6098c2c4ae0e97e78f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/237ccaa45bca4b6098c2c4ae0e97e78f 2023-04-07 09:55:40,828 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:40,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:40,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41dc2033[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:40,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:40,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,833 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,833 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:40,833 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:40,833 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:40,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:40,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:40,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:40,835 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:40,835 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/d932944dea6d436682b81b11e41bc09a 2023-04-07 09:55:40,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:40,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:40,839 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:40,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:40,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:40,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:40,839 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:40,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,840 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:40,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,843 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 237ccaa45bca4b6098c2c4ae0e97e78f(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:40,843 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:40,843 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861340751; duration=0sec 2023-04-07 09:55:40,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861400800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,843 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:40,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861400809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,844 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,845 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:40,845 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:40,845 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,846 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81bfff3eb42d4a87bbe3c947947c619d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8764bd7e6cea48a4adf6c0520921adaf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8f0d6af560f442193902223365dcbde] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=35.7 K 2023-04-07 09:55:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,846 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 81bfff3eb42d4a87bbe3c947947c619d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1680861338524 2023-04-07 09:55:40,847 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8764bd7e6cea48a4adf6c0520921adaf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1680861338870 2023-04-07 09:55:40,847 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e8f0d6af560f442193902223365dcbde, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680861340006 2023-04-07 09:55:40,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861400844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861400844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861400844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:40,853 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:40,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:40,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:40,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:40,853 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:40,854 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:40,853 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:40,860 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:40,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861400945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861400945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,954 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:40,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:40,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:40,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:40,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:40,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861400953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,957 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:40,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:40,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:40,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:40,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861400953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:40,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861400953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,958 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,959 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:40,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:40,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,959 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:40,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:40,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:40,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:40,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:40,961 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:40,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:40,962 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,962 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:40,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:40,962 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:40,962 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:40,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:40,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:40,962 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:40,963 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:40,965 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,972 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:40,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:40,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@506badaa[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:40,975 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:40,975 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,980 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:40,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:40,980 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:40,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:40,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:40,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:40,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:40,986 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:40,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:40,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:40,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:40,987 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:40,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:40,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:40,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:40,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,994 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:40,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:40,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:40,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:40,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:40,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:40,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:40,998 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:40,998 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:40,998 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:40,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:41,098 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:41,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:41,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:41,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:41,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:41,105 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:41,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:41,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:41,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:41,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:41,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:41,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:41,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:41,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:41,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:41,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:41,115 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:41,116 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:41,116 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:41,116 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:41,116 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:41,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:41,118 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:41,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,118 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:41,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f3692[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:41,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:41,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,120 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:41,121 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,122 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:41,123 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,123 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:41,123 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:41,123 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:41,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:41,127 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:41,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:41,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:41,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:41,127 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:41,134 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,134 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,134 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:41,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:41,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,138 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:41,138 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:41,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:41,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:41,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:41,138 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:41,138 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:41,138 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,139 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,139 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,139 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,139 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,139 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:41,139 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:41,139 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:41,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:41,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861401151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861401151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861401158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861401159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861401159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,200 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/a743115b00ac498e8dad284aae91a361 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a743115b00ac498e8dad284aae91a361 2023-04-07 09:55:41,204 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into a743115b00ac498e8dad284aae91a361(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:41,204 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:41,204 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861340751; duration=0sec 2023-04-07 09:55:41,204 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:41,239 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:41,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:41,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:41,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:41,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:41,249 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:41,249 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:41,249 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:41,249 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:41,249 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,249 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:41,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:41,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:41,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:41,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:41,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:41,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:41,259 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:41,260 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:41,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/34c47edef79b4680a929bdc2d5a20517 2023-04-07 09:55:41,264 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:41,264 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:41,264 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:41,264 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:41,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:41,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:41,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:41,264 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:41,266 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,266 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:41,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,267 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d488696[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:41,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:41,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,270 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:41,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:41,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/18b10f8de2374b10a9e3dd3b12356cbd 2023-04-07 09:55:41,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:41,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:41,280 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:41,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:41,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:41,280 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:41,280 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:41,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:41,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/d932944dea6d436682b81b11e41bc09a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d932944dea6d436682b81b11e41bc09a 2023-04-07 09:55:41,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:41,286 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2f3662b2d1394bbf8f3ae7fcb6896ec3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2f3662b2d1394bbf8f3ae7fcb6896ec3 2023-04-07 09:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d932944dea6d436682b81b11e41bc09a, entries=200, sequenceid=235, filesize=14.1 K 2023-04-07 09:55:41,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:41,289 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:41,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:41,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:41,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:41,291 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:41,291 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:41,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:41,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/34c47edef79b4680a929bdc2d5a20517 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/34c47edef79b4680a929bdc2d5a20517 2023-04-07 09:55:41,294 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 2f3662b2d1394bbf8f3ae7fcb6896ec3(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:41,294 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:41,294 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861340751; duration=0sec 2023-04-07 09:55:41,294 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:41,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/34c47edef79b4680a929bdc2d5a20517, entries=150, sequenceid=235, filesize=11.8 K 2023-04-07 09:55:41,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/18b10f8de2374b10a9e3dd3b12356cbd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/18b10f8de2374b10a9e3dd3b12356cbd 2023-04-07 09:55:41,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/18b10f8de2374b10a9e3dd3b12356cbd, entries=150, sequenceid=235, filesize=11.8 K 2023-04-07 09:55:41,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for c05568a99c8bbb1b0521e425f41a54f2 in 522ms, sequenceid=235, compaction requested=false 2023-04-07 09:55:41,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:41,391 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:41,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:41,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:41,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:41,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:41,395 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:41,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:41,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:41,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:41,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,395 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,397 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:41,397 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:41,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:41,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:41,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:41,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:41,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:41,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:41,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:41,399 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:41,399 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:41,407 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/b49e566e0b034ee9a4f5400c3b0399cb 2023-04-07 09:55:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:41,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:41,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861401472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861401474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861401478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861401480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861401481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861401586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861401586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861401586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861401588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861401589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861401790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861401791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861401792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861401792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:41,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861401792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:41,817 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/2a5111cf93be4827a1f6b61fc72c4341 2023-04-07 09:55:42,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861402094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861402096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861402098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861402098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861402099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,226 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/b2823b7cebf246a9a68c32616537cf41 2023-04-07 09:55:42,230 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/b49e566e0b034ee9a4f5400c3b0399cb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b49e566e0b034ee9a4f5400c3b0399cb 2023-04-07 09:55:42,233 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b49e566e0b034ee9a4f5400c3b0399cb, entries=150, sequenceid=254, filesize=11.8 K 2023-04-07 09:55:42,234 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/2a5111cf93be4827a1f6b61fc72c4341 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/2a5111cf93be4827a1f6b61fc72c4341 2023-04-07 09:55:42,236 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/2a5111cf93be4827a1f6b61fc72c4341, entries=150, sequenceid=254, filesize=11.8 K 2023-04-07 09:55:42,237 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/b2823b7cebf246a9a68c32616537cf41 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b2823b7cebf246a9a68c32616537cf41 2023-04-07 09:55:42,240 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b2823b7cebf246a9a68c32616537cf41, entries=150, sequenceid=254, filesize=11.8 K 2023-04-07 09:55:42,240 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c05568a99c8bbb1b0521e425f41a54f2 in 841ms, sequenceid=254, compaction requested=true 2023-04-07 09:55:42,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:42,240 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:42,240 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:42,241 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:42,241 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,241 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:42,241 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:42,242 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,242 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:42,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:42,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db5feb4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:42,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:42,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,245 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:42,245 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:42,245 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:42,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,245 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:42,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:42,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:42,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:42,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:42,254 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:42,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:42,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:42,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:42,254 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:42,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,255 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:42,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,255 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:42,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:42,258 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,259 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,259 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:42,259 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:42,259 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:42,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:42,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:42,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:42,260 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:42,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:42,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:42,260 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,260 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:42,260 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:42,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:42,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:42,360 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:42,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:42,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:42,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:42,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:42,363 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:42,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:42,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:42,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:42,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:42,364 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,365 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:42,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:42,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:42,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:42,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:42,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:42,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:42,368 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:42,368 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-07 09:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:42,368 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:42,369 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:42,369 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:42,369 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:42,394 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/306a61aa697d45d58cb04813ffa139be 2023-04-07 09:55:42,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:42,604 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:42,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861402614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861402618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861402618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861402619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861402620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861402721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861402725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861402725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861402727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861402727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,806 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/f2719870af17455b86bb0f66ea4dfe44 2023-04-07 09:55:42,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861402928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861402930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861402930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861402930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:42,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49496 deadline: 1680861402930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,217 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa 2023-04-07 09:55:43,224 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/306a61aa697d45d58cb04813ffa139be as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/306a61aa697d45d58cb04813ffa139be 2023-04-07 09:55:43,230 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/306a61aa697d45d58cb04813ffa139be, entries=150, sequenceid=270, filesize=11.9 K 2023-04-07 09:55:43,231 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/f2719870af17455b86bb0f66ea4dfe44 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f2719870af17455b86bb0f66ea4dfe44 2023-04-07 09:55:43,238 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f2719870af17455b86bb0f66ea4dfe44, entries=150, sequenceid=270, filesize=11.9 K 2023-04-07 09:55:43,239 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa 2023-04-07 09:55:43,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861403233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861403234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861403236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,242 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa, entries=150, sequenceid=270, filesize=11.9 K 2023-04-07 09:55:43,243 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c05568a99c8bbb1b0521e425f41a54f2 in 875ms, sequenceid=270, compaction requested=true 2023-04-07 09:55:43,243 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:43,243 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:43,243 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:43,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:43,246 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-07 09:55:43,246 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:43,246 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,247 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:43,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:43,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:43,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:43,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:43,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403e1c55[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:43,249 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:43,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,251 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,251 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:43,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:43,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:43,251 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:43,261 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:43,261 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,261 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:43,263 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:43,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:43,268 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:43,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:43,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:43,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:43,268 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:43,270 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,270 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,270 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,274 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,279 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:43,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:43,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:43,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,279 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:43,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,280 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,280 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,280 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,279 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:43,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49554 deadline: 1680861403295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861403299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,380 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:43,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:43,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:43,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:43,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:43,383 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:43,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:43,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:43,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,383 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,384 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,384 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,385 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,385 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:43,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:43,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:43,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:43,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:43,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:43,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:43,386 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,386 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:43,386 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:43,386 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,386 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:43,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:43,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:43,387 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:43,388 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:43,390 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:43,390 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fd0966[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:43,391 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:43,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:43,391 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,391 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:43,391 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:43,394 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,394 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:43,394 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,394 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:43,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:43,398 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:43,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:43,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:43,398 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:43,398 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:43,401 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,401 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,401 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,402 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,404 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:43,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:43,406 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:43,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,407 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:43,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,407 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:43,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861403404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861403404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,507 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:43,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:43,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:43,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:43,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:43,513 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:43,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:43,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:43,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,513 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,514 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,514 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:43,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:43,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:43,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,522 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:43,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112844d9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:43,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,525 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:43,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:43,525 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,525 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:43,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:43,526 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:43,526 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,526 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:43,527 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:43,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:43,530 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:43,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:43,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:43,530 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:43,530 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:43,531 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,531 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,531 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,544 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:43,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,544 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:43,544 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,545 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861403609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861403610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,645 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:43,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:43,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:43,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:43,648 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:43,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:43,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:43,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,649 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,649 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:43,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:43,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:43,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:43,651 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:43,651 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,653 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:43,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:43,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730a4816[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:43,657 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:43,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:43,661 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:43,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,661 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:43,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:43,664 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:43,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:43,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:43,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:43,664 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:43,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,665 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,666 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,667 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,669 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,669 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,669 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,669 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:43,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:43,669 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,669 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,670 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:43,670 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:43,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:43,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/778b0b0f4d0d45fa9af183ed29e800a9 2023-04-07 09:55:43,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1680861403741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49482 deadline: 1680861403741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49492 deadline: 1680861403744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,770 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:43,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:43,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:43,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:43,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:43,774 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:43,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:43,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:43,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,775 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,775 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,776 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:43,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:43,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:43,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:43,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:43,782 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,783 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,783 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:43,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:43,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43663e96[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:43,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,785 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,785 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:43,785 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:43,785 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,786 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:43,785 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:43,787 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:43,787 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,787 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:43,788 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:43,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:43,790 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:43,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:43,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:43,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:43,790 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:43,791 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,791 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,791 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,793 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,796 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:43,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:43,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:43,796 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,796 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,897 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:43,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:43,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:43,902 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:43,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:43,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:43,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:43,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:43,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:43,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:43,905 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:43,905 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:43,907 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:43,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1300b9a5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:43,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:43,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,910 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:43,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:43,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:43,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:43,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,913 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:43,913 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,913 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:43,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:43,916 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:43,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:43,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:43,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:43,916 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:43,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:43,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:43,919 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,920 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861403914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:43,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861403915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:43,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:43,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:43,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:43,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:43,925 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:43,926 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:43,926 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:43,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,026 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:44,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:44,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:44,029 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:44,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:44,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:44,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,031 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,031 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:44,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:44,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:44,033 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,033 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:44,033 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:44,034 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,035 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:44,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:44,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40dd2990[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:44,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:44,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:44,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:44,039 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:44,041 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:44,042 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:44,042 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:44,042 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:44,042 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:44,044 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,044 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,045 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,045 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,048 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:44,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:44,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:44,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,049 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:44,049 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:44,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/62c57bbee2314385ad9203faec28d4cc 2023-04-07 09:55:44,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/4133f05a28bb4e37b9686c6806468538 2023-04-07 09:55:44,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/778b0b0f4d0d45fa9af183ed29e800a9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/778b0b0f4d0d45fa9af183ed29e800a9 2023-04-07 09:55:44,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/778b0b0f4d0d45fa9af183ed29e800a9, entries=200, sequenceid=288, filesize=14.3 K 2023-04-07 09:55:44,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/62c57bbee2314385ad9203faec28d4cc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/62c57bbee2314385ad9203faec28d4cc 2023-04-07 09:55:44,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/62c57bbee2314385ad9203faec28d4cc, entries=150, sequenceid=288, filesize=11.9 K 2023-04-07 09:55:44,131 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/4133f05a28bb4e37b9686c6806468538 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/4133f05a28bb4e37b9686c6806468538 2023-04-07 09:55:44,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/4133f05a28bb4e37b9686c6806468538, entries=150, sequenceid=288, filesize=11.9 K 2023-04-07 09:55:44,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c05568a99c8bbb1b0521e425f41a54f2 in 889ms, sequenceid=288, compaction requested=true 2023-04-07 09:55:44,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:44,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:44,134 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:55:44,134 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:55:44,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:44,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:44,135 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 65413 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:55:44,136 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:44,136 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,136 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/237ccaa45bca4b6098c2c4ae0e97e78f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d932944dea6d436682b81b11e41bc09a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b49e566e0b034ee9a4f5400c3b0399cb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/306a61aa697d45d58cb04813ffa139be, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/778b0b0f4d0d45fa9af183ed29e800a9] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=63.9 K 2023-04-07 09:55:44,136 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:55:44,136 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 237ccaa45bca4b6098c2c4ae0e97e78f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680861340006 2023-04-07 09:55:44,136 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:44,136 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,136 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a743115b00ac498e8dad284aae91a361, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/34c47edef79b4680a929bdc2d5a20517, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/2a5111cf93be4827a1f6b61fc72c4341, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f2719870af17455b86bb0f66ea4dfe44, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/62c57bbee2314385ad9203faec28d4cc] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=59.2 K 2023-04-07 09:55:44,136 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d932944dea6d436682b81b11e41bc09a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680861340661 2023-04-07 09:55:44,137 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a743115b00ac498e8dad284aae91a361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680861340006 2023-04-07 09:55:44,137 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b49e566e0b034ee9a4f5400c3b0399cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680861340794 2023-04-07 09:55:44,137 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 34c47edef79b4680a929bdc2d5a20517, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680861340672 2023-04-07 09:55:44,137 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 306a61aa697d45d58cb04813ffa139be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680861341469 2023-04-07 09:55:44,137 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2a5111cf93be4827a1f6b61fc72c4341, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680861340794 2023-04-07 09:55:44,137 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f2719870af17455b86bb0f66ea4dfe44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680861341469 2023-04-07 09:55:44,137 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 778b0b0f4d0d45fa9af183ed29e800a9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680861342616 2023-04-07 09:55:44,138 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 62c57bbee2314385ad9203faec28d4cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680861342616 2023-04-07 09:55:44,146 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#509 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:44,147 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#510 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:44,149 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:44,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:44,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:44,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:44,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:44,152 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:44,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:44,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:44,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,155 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,155 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,155 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:44,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:44,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:44,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,160 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:44,161 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 09:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:44,161 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:44,434 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:44,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861404482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861404489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,557 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/c585c5c35f804360a6f17fbad6444ab2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c585c5c35f804360a6f17fbad6444ab2 2023-04-07 09:55:44,561 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/43de347be9e6412aaa27be6c54358d5d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/43de347be9e6412aaa27be6c54358d5d 2023-04-07 09:55:44,562 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into c585c5c35f804360a6f17fbad6444ab2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:44,562 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:44,562 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=11, startTime=1680861344134; duration=0sec 2023-04-07 09:55:44,562 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:44,562 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:55:44,563 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:55:44,563 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:44,563 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,563 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2f3662b2d1394bbf8f3ae7fcb6896ec3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/18b10f8de2374b10a9e3dd3b12356cbd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b2823b7cebf246a9a68c32616537cf41, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/4133f05a28bb4e37b9686c6806468538] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=59.2 K 2023-04-07 09:55:44,564 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 2f3662b2d1394bbf8f3ae7fcb6896ec3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680861340006 2023-04-07 09:55:44,564 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 18b10f8de2374b10a9e3dd3b12356cbd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680861340672 2023-04-07 09:55:44,564 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b2823b7cebf246a9a68c32616537cf41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680861340794 2023-04-07 09:55:44,564 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into 43de347be9e6412aaa27be6c54358d5d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:44,564 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:44,564 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=11, startTime=1680861344134; duration=0sec 2023-04-07 09:55:44,565 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:44,565 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 76d0c6d3c3354d6bb7a6a2a4aec7f3aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680861341469 2023-04-07 09:55:44,565 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4133f05a28bb4e37b9686c6806468538, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680861342616 2023-04-07 09:55:44,570 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/4f1085d336304924ad0fe851eab8b3a7 2023-04-07 09:55:44,572 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#512 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:44,578 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/06f3350e9dbd4a75889dd97c2f1969dc 2023-04-07 09:55:44,583 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/ee3eb0ebe0f040ada4aede7126a96c3c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ee3eb0ebe0f040ada4aede7126a96c3c 2023-04-07 09:55:44,588 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e8811fde885043e2a3eb2cf98bf8137d 2023-04-07 09:55:44,588 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into ee3eb0ebe0f040ada4aede7126a96c3c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:44,588 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:44,588 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=11, startTime=1680861344134; duration=0sec 2023-04-07 09:55:44,588 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:44,591 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/4f1085d336304924ad0fe851eab8b3a7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4f1085d336304924ad0fe851eab8b3a7 2023-04-07 09:55:44,594 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4f1085d336304924ad0fe851eab8b3a7, entries=150, sequenceid=302, filesize=11.9 K 2023-04-07 09:55:44,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861404590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,595 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/06f3350e9dbd4a75889dd97c2f1969dc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/06f3350e9dbd4a75889dd97c2f1969dc 2023-04-07 09:55:44,599 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/06f3350e9dbd4a75889dd97c2f1969dc, entries=150, sequenceid=302, filesize=11.9 K 2023-04-07 09:55:44,599 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e8811fde885043e2a3eb2cf98bf8137d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8811fde885043e2a3eb2cf98bf8137d 2023-04-07 09:55:44,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861404597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,602 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8811fde885043e2a3eb2cf98bf8137d, entries=150, sequenceid=302, filesize=11.9 K 2023-04-07 09:55:44,603 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c05568a99c8bbb1b0521e425f41a54f2 in 443ms, sequenceid=302, compaction requested=false 2023-04-07 09:55:44,603 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:44,603 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:44,603 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:44,604 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,604 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:44,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61369365[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:44,606 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:44,606 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,608 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:44,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:44,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:44,608 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,608 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:44,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,610 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:44,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:44,612 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:44,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:44,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:44,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:44,612 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:44,613 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,613 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,613 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:44,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:44,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:44,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,617 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:44,617 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:44,618 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,618 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,717 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:44,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:44,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:44,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:44,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:44,720 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:44,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:44,720 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:44,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,723 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:44,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:44,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:44,725 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,725 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:44,725 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-07 09:55:44,725 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:44,725 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,725 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:44,726 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,726 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:44,726 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,735 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/887a4bbaf32845cdad766da0646f3873 2023-04-07 09:55:44,747 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/04856b55ec0541949b42cc60d70aa23b 2023-04-07 09:55:44,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:44,755 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:44,757 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2bc6e330408044c9ade5c689d4a69a6b 2023-04-07 09:55:44,760 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/887a4bbaf32845cdad766da0646f3873 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/887a4bbaf32845cdad766da0646f3873 2023-04-07 09:55:44,763 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/887a4bbaf32845cdad766da0646f3873, entries=150, sequenceid=323, filesize=11.9 K 2023-04-07 09:55:44,764 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/04856b55ec0541949b42cc60d70aa23b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/04856b55ec0541949b42cc60d70aa23b 2023-04-07 09:55:44,768 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/04856b55ec0541949b42cc60d70aa23b, entries=150, sequenceid=323, filesize=11.9 K 2023-04-07 09:55:44,769 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2bc6e330408044c9ade5c689d4a69a6b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2bc6e330408044c9ade5c689d4a69a6b 2023-04-07 09:55:44,772 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2bc6e330408044c9ade5c689d4a69a6b, entries=150, sequenceid=323, filesize=11.9 K 2023-04-07 09:55:44,773 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=60.38 KB/61830 for c05568a99c8bbb1b0521e425f41a54f2 in 48ms, sequenceid=323, compaction requested=true 2023-04-07 09:55:44,773 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:44,773 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:44,773 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:44,774 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:44,776 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:55:44,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:44,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:44,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,777 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:44,777 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:44,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d6935b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:44,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:44,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,779 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:44,779 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:44,779 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:44,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,781 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:44,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:44,784 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:44,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:44,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:44,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:44,784 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:44,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,788 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:44,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:44,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:44,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,792 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:44,792 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:44,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,792 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861404803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861404804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861404806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861404810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861404810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,892 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:44,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:44,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:44,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:44,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:44,895 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:44,895 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,895 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:44,895 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:44,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,896 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,899 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,899 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:44,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:44,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:44,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:44,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:44,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:44,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,903 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:44,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ddd15[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:44,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:44,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,907 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:44,908 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,908 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:44,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:44,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,909 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:44,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:44,912 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:44,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:44,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:44,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:44,912 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:44,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:44,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:44,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:44,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861404911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861404912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861404912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861404914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:44,920 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:44,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:44,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:44,920 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:44,920 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:44,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:44,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:44,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:44,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:44,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861404917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,020 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,023 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cb5e59d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,031 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,031 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,034 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,036 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,036 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,039 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,041 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,041 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,042 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,042 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861405118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861405118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861405118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861405121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861405123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,142 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,145 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,145 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,146 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,146 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,147 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,147 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,148 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,148 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,148 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,148 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,150 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@794de6d5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,152 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,153 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,153 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,155 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,157 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,157 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,157 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,157 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,158 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,159 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,159 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,165 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:45,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,165 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/99c1c6aa203b4481bdab87c114e9002e 2023-04-07 09:55:45,265 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,268 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,268 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,270 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,276 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,276 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,278 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e26feec[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,278 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,280 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,281 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,281 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,282 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,282 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,282 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,282 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,285 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,285 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,285 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,286 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,288 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,290 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,290 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,290 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:45,391 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,396 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,396 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,397 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,399 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,399 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,404 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,404 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,405 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,405 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,406 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,407 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,407 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,408 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3528f548[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,408 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,409 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,409 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,409 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,409 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,409 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,409 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,412 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,422 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,422 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,426 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,431 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,431 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861405425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,431 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,431 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,431 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861405425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861405426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,433 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,433 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861405426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861405427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,533 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,536 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,537 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,537 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,538 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,538 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,539 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,539 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,539 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,539 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,542 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,543 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b2777d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,547 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,547 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,551 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,551 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,556 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,557 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,557 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:45,557 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,557 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,557 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,557 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/aa22b3ad7f494beaa047cfb9cb4b1653 2023-04-07 09:55:45,657 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,661 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,661 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,661 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,661 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,661 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,661 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,664 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,664 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,666 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,666 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,666 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,666 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,668 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@205f5430[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,669 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,669 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,671 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,675 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,675 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,678 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,678 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,678 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,679 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,679 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,679 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,683 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:45,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,683 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,683 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,684 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,684 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,784 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,787 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,789 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49671f49[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,794 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,794 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,795 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,796 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,796 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,796 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,799 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,799 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,804 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,806 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,808 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,808 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,808 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,909 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:45,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:45,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:45,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:45,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:45,912 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:45,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:45,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:45,913 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,913 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,914 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:45,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:45,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:45,920 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,920 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:45,920 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:45,920 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:45,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:45,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:45,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:45,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:45,922 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,922 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,923 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b173198[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:45,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:45,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,930 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,930 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,930 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:45,930 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:45,931 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:45,931 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:45,936 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,936 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:45,936 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,937 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:45,941 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:45,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:45,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:45,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:45,941 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:45,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:45,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:45,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861405936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861405937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:45,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861405937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861405937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:45,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861405937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:45,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:45,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:45,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:45,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:45,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:45,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:45,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:45,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:45,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:45,948 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:45,948 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:45,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:46,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/46075c6648e24fbb82538b3db992f27c 2023-04-07 09:55:46,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/99c1c6aa203b4481bdab87c114e9002e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/99c1c6aa203b4481bdab87c114e9002e 2023-04-07 09:55:46,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/99c1c6aa203b4481bdab87c114e9002e, entries=200, sequenceid=338, filesize=14.3 K 2023-04-07 09:55:46,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/aa22b3ad7f494beaa047cfb9cb4b1653 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/aa22b3ad7f494beaa047cfb9cb4b1653 2023-04-07 09:55:46,011 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/aa22b3ad7f494beaa047cfb9cb4b1653, entries=150, sequenceid=338, filesize=11.9 K 2023-04-07 09:55:46,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/46075c6648e24fbb82538b3db992f27c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/46075c6648e24fbb82538b3db992f27c 2023-04-07 09:55:46,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/46075c6648e24fbb82538b3db992f27c, entries=150, sequenceid=338, filesize=11.9 K 2023-04-07 09:55:46,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c05568a99c8bbb1b0521e425f41a54f2 in 1245ms, sequenceid=338, compaction requested=true 2023-04-07 09:55:46,021 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:46,021 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:46,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:46,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:55:46,022 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:46,023 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:46,023 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:46,023 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,023 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/43de347be9e6412aaa27be6c54358d5d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/06f3350e9dbd4a75889dd97c2f1969dc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/04856b55ec0541949b42cc60d70aa23b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/aa22b3ad7f494beaa047cfb9cb4b1653] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=47.9 K 2023-04-07 09:55:46,023 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51458 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:46,023 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 43de347be9e6412aaa27be6c54358d5d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680861342616 2023-04-07 09:55:46,023 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:46,024 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,024 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c585c5c35f804360a6f17fbad6444ab2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4f1085d336304924ad0fe851eab8b3a7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/887a4bbaf32845cdad766da0646f3873, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/99c1c6aa203b4481bdab87c114e9002e] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=50.3 K 2023-04-07 09:55:46,024 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 06f3350e9dbd4a75889dd97c2f1969dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680861343283 2023-04-07 09:55:46,024 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting c585c5c35f804360a6f17fbad6444ab2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680861342616 2023-04-07 09:55:46,024 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 04856b55ec0541949b42cc60d70aa23b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680861344456 2023-04-07 09:55:46,024 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4f1085d336304924ad0fe851eab8b3a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680861343283 2023-04-07 09:55:46,024 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting aa22b3ad7f494beaa047cfb9cb4b1653, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680861344775 2023-04-07 09:55:46,024 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 887a4bbaf32845cdad766da0646f3873, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680861344456 2023-04-07 09:55:46,025 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 99c1c6aa203b4481bdab87c114e9002e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680861344754 2023-04-07 09:55:46,032 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#521 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:46,034 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:46,040 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/c9c7d31353ba4235adfee6cddac9dd6b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c9c7d31353ba4235adfee6cddac9dd6b 2023-04-07 09:55:46,044 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into c9c7d31353ba4235adfee6cddac9dd6b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:46,044 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,044 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=12, startTime=1680861346021; duration=0sec 2023-04-07 09:55:46,044 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:46,044 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:46,045 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:46,045 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:46,045 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,045 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ee3eb0ebe0f040ada4aede7126a96c3c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8811fde885043e2a3eb2cf98bf8137d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2bc6e330408044c9ade5c689d4a69a6b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/46075c6648e24fbb82538b3db992f27c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=47.9 K 2023-04-07 09:55:46,046 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ee3eb0ebe0f040ada4aede7126a96c3c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1680861342616 2023-04-07 09:55:46,046 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e8811fde885043e2a3eb2cf98bf8137d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680861343283 2023-04-07 09:55:46,046 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2bc6e330408044c9ade5c689d4a69a6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680861344456 2023-04-07 09:55:46,046 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 46075c6648e24fbb82538b3db992f27c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680861344775 2023-04-07 09:55:46,048 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:46,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:46,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:46,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:46,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:46,055 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:46,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:46,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:46,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,056 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,057 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#523 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:46,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:46,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,058 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:46,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:46,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:46,060 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,060 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:46,060 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-07 09:55:46,060 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:46,060 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:46,061 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:46,061 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:46,061 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:46,061 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:46,078 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/ccd89302788845828143db0c82804c67 2023-04-07 09:55:46,097 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/f572f5785dc8420fa6abfa0aaa21c939 2023-04-07 09:55:46,442 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/0589d37d82ed4f1b9e29d2cdbb4da5da as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/0589d37d82ed4f1b9e29d2cdbb4da5da 2023-04-07 09:55:46,447 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 0589d37d82ed4f1b9e29d2cdbb4da5da(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:46,447 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,447 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=12, startTime=1680861346021; duration=0sec 2023-04-07 09:55:46,447 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:46,469 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2871621eac674036850ffd4bfe5d8bad as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2871621eac674036850ffd4bfe5d8bad 2023-04-07 09:55:46,473 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 2871621eac674036850ffd4bfe5d8bad(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:46,473 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,473 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=12, startTime=1680861346022; duration=0sec 2023-04-07 09:55:46,473 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:46,506 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/7e4626c1a3cf4ff39773d875269a0600 2023-04-07 09:55:46,510 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/ccd89302788845828143db0c82804c67 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ccd89302788845828143db0c82804c67 2023-04-07 09:55:46,513 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ccd89302788845828143db0c82804c67, entries=150, sequenceid=356, filesize=11.9 K 2023-04-07 09:55:46,514 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/f572f5785dc8420fa6abfa0aaa21c939 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f572f5785dc8420fa6abfa0aaa21c939 2023-04-07 09:55:46,518 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f572f5785dc8420fa6abfa0aaa21c939, entries=150, sequenceid=356, filesize=11.9 K 2023-04-07 09:55:46,519 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/7e4626c1a3cf4ff39773d875269a0600 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7e4626c1a3cf4ff39773d875269a0600 2023-04-07 09:55:46,523 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7e4626c1a3cf4ff39773d875269a0600, entries=150, sequenceid=356, filesize=11.9 K 2023-04-07 09:55:46,523 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for c05568a99c8bbb1b0521e425f41a54f2 in 463ms, sequenceid=356, compaction requested=false 2023-04-07 09:55:46,523 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,523 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:46,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,525 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,525 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3ba018[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:46,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:46,527 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,528 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:46,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,528 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:46,528 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:46,530 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:46,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,530 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:46,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:46,533 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:46,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:46,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:46,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:46,533 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:46,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,541 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:46,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,542 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:46,542 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:46,642 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:46,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:46,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:46,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:46,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:46,645 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:46,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:46,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:46,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,645 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,646 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,647 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,647 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:46,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,648 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,654 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:46,654 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:46,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:46,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@441f893b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:46,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,660 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:46,660 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,660 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:46,662 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:46,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:46,665 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:46,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:46,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:46,665 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:46,665 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:46,666 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,666 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,666 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,667 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,668 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:46,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:46,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:46,670 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,671 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:46,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,671 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,671 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,774 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:46,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:46,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:46,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:46,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:46,777 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:46,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:46,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:46,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,777 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,780 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:46,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,781 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:46,782 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,782 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:46,782 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,782 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:46,784 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,784 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:46,785 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35735ba6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:46,785 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:46,787 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:46,787 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:46,788 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,789 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:46,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:46,792 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:46,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:46,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:46,792 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:46,792 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:46,795 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,795 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,795 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,795 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,799 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,799 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,800 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:46,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:46,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,800 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:46,800 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:46,801 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:46,901 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:46,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:46,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:46,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:46,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:46,904 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:46,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:46,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:46,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,905 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:46,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:46,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:46,907 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,907 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:46,907 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:46,907 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:46,907 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:46,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,909 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:46,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@376a3761[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:46,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:46,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:46,914 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:46,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:46,916 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:46,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:46,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:46,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:46,916 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:46,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:46,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:46,918 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,920 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:46,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:46,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:46,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:46,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:46,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:46,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:46,929 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:46,929 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:46,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:46,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:46,956 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:46,957 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:46,957 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:46,957 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:46,957 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:46,957 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:46,957 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:46,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861346979, max timestamp=1680861346988Num uniques 859; 2023-04-07 09:55:46,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861346979, max timestamp=1680861346988Num uniques 859; 2023-04-07 09:55:46,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:55:46,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861346979, max timestamp=1680861346988Num uniques 859; 2023-04-07 09:55:46,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:55:46,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:55:47,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861407005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861407005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861407009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861407010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861407010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,029 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:47,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:47,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:47,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:47,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:47,035 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:47,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:47,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:47,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:47,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:47,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:47,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:47,044 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,044 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:47,044 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:47,044 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:47,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:47,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74927af4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:47,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,051 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:47,051 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:47,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:47,052 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,052 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:47,052 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,052 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:47,055 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:47,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:47,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:47,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:47,055 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:47,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:47,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:47,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,062 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:47,062 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,062 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:47,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861407113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861407113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861407113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861407117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861407118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,162 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:47,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:47,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:47,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:47,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:47,166 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:47,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:47,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:47,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,166 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,168 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:47,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:47,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:47,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:47,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:47,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cfc730c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:47,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:47,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:47,176 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:47,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:47,178 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:47,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:47,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:47,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:47,178 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:47,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:47,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,192 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:47,192 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:47,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,292 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:47,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:47,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:47,296 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:47,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:47,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:47,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,298 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:47,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:47,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:47,300 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,301 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:47,301 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:47,301 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,301 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:47,303 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:47,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63fdbf5e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:47,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,307 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:47,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:47,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:47,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:47,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:47,314 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:47,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:47,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:47,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:47,314 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:47,315 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,315 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,315 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,316 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:47,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:47,321 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:47,322 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:47,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,322 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861407320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861407320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861407320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861407325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861407325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/39c1ae15e49f424aac159b96a6f784ac 2023-04-07 09:55:47,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/553a5675dc7c4dd0a3db0206bd455a1f 2023-04-07 09:55:47,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/597fffac677848ab97bb89d9ef8e7db1 2023-04-07 09:55:47,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/39c1ae15e49f424aac159b96a6f784ac as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/39c1ae15e49f424aac159b96a6f784ac 2023-04-07 09:55:47,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/39c1ae15e49f424aac159b96a6f784ac, entries=200, sequenceid=372, filesize=14.3 K 2023-04-07 09:55:47,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/553a5675dc7c4dd0a3db0206bd455a1f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/553a5675dc7c4dd0a3db0206bd455a1f 2023-04-07 09:55:47,399 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/553a5675dc7c4dd0a3db0206bd455a1f, entries=150, sequenceid=372, filesize=11.9 K 2023-04-07 09:55:47,400 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/597fffac677848ab97bb89d9ef8e7db1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/597fffac677848ab97bb89d9ef8e7db1 2023-04-07 09:55:47,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/597fffac677848ab97bb89d9ef8e7db1, entries=150, sequenceid=372, filesize=11.9 K 2023-04-07 09:55:47,407 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c05568a99c8bbb1b0521e425f41a54f2 in 451ms, sequenceid=372, compaction requested=true 2023-04-07 09:55:47,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:47,407 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:47,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:47,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:47,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:47,408 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:47,409 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39408 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:47,409 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:47,409 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,409 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/0589d37d82ed4f1b9e29d2cdbb4da5da, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ccd89302788845828143db0c82804c67, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/39c1ae15e49f424aac159b96a6f784ac] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=38.5 K 2023-04-07 09:55:47,409 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:47,410 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:47,410 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 0589d37d82ed4f1b9e29d2cdbb4da5da, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680861344775 2023-04-07 09:55:47,410 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,410 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c9c7d31353ba4235adfee6cddac9dd6b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f572f5785dc8420fa6abfa0aaa21c939, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/553a5675dc7c4dd0a3db0206bd455a1f] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.1 K 2023-04-07 09:55:47,410 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ccd89302788845828143db0c82804c67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861344794 2023-04-07 09:55:47,410 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c9c7d31353ba4235adfee6cddac9dd6b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680861344775 2023-04-07 09:55:47,410 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 39c1ae15e49f424aac159b96a6f784ac, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680861346951 2023-04-07 09:55:47,410 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f572f5785dc8420fa6abfa0aaa21c939, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861344794 2023-04-07 09:55:47,411 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 553a5675dc7c4dd0a3db0206bd455a1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680861346954 2023-04-07 09:55:47,422 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:47,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:47,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:47,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:47,429 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:47,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:47,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:47,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,433 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,433 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,434 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,434 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:47,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:47,435 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:47,444 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,444 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:47,444 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-07 09:55:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:47,446 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:47,499 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/dceafe5486794c5db274b3dc2ac88512 2023-04-07 09:55:47,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:47,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:47,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861407640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861407640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861407642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861407643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861407644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861407745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861407750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861407750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861407753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:47,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861407753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,858 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/bbbe3f036c604ea88687442ac5bb32cd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/bbbe3f036c604ea88687442ac5bb32cd 2023-04-07 09:55:47,862 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into bbbe3f036c604ea88687442ac5bb32cd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:47,862 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:47,862 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861347408; duration=0sec 2023-04-07 09:55:47,862 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:47,862 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:47,863 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:47,863 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:47,863 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,864 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2871621eac674036850ffd4bfe5d8bad, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7e4626c1a3cf4ff39773d875269a0600, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/597fffac677848ab97bb89d9ef8e7db1] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.1 K 2023-04-07 09:55:47,864 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2871621eac674036850ffd4bfe5d8bad, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680861344775 2023-04-07 09:55:47,864 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7e4626c1a3cf4ff39773d875269a0600, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680861344794 2023-04-07 09:55:47,864 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 597fffac677848ab97bb89d9ef8e7db1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680861346954 2023-04-07 09:55:47,869 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:47,891 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/675384180f944d6e947d1d1c6689f733 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/675384180f944d6e947d1d1c6689f733 2023-04-07 09:55:47,894 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 675384180f944d6e947d1d1c6689f733(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:47,894 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:47,894 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861347407; duration=0sec 2023-04-07 09:55:47,895 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:47,911 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/c4cb2fbe0d154aa1a757397f0334af99 2023-04-07 09:55:47,923 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/b3139898f7564b949c3ea5953e26b59e 2023-04-07 09:55:47,931 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/dceafe5486794c5db274b3dc2ac88512 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/dceafe5486794c5db274b3dc2ac88512 2023-04-07 09:55:47,937 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/dceafe5486794c5db274b3dc2ac88512, entries=150, sequenceid=407, filesize=11.9 K 2023-04-07 09:55:47,937 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/c4cb2fbe0d154aa1a757397f0334af99 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c4cb2fbe0d154aa1a757397f0334af99 2023-04-07 09:55:47,941 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c4cb2fbe0d154aa1a757397f0334af99, entries=150, sequenceid=407, filesize=11.9 K 2023-04-07 09:55:47,942 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/b3139898f7564b949c3ea5953e26b59e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b3139898f7564b949c3ea5953e26b59e 2023-04-07 09:55:47,953 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b3139898f7564b949c3ea5953e26b59e, entries=150, sequenceid=407, filesize=11.9 K 2023-04-07 09:55:47,953 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for c05568a99c8bbb1b0521e425f41a54f2 in 509ms, sequenceid=407, compaction requested=false 2023-04-07 09:55:47,953 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:47,953 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:47,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:47,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:47,954 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:47,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:47,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:55:47,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:47,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:47,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:47,958 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:47,958 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,959 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:47,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca4d7f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:47,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:47,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:47,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:47,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:47,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:47,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:47,980 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:47,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:47,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:47,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:47,981 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:47,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:47,985 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:47,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:47,989 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:47,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:47,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,989 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:47,989 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:47,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:47,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:47,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861407995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861407996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861407996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861407997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861407998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,089 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,092 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,094 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,097 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6268d2e8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,099 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,101 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,102 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,105 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,105 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,105 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861408107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,114 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861408110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,114 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,115 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,115 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861408110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,116 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861408110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861408110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,215 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,218 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,218 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,218 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,218 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,218 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,218 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,221 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,223 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,223 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,223 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,224 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da3b9ba[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,226 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,227 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,230 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,230 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,232 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,237 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,237 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,237 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,280 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/d45acbd8456b40cfa09dbad872cb4b72 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d45acbd8456b40cfa09dbad872cb4b72 2023-04-07 09:55:48,285 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into d45acbd8456b40cfa09dbad872cb4b72(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:48,285 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:48,285 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861347408; duration=0sec 2023-04-07 09:55:48,285 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:48,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861408315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861408315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861408316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861408317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861408318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,337 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,340 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,341 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,341 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,342 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,344 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,347 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43aaf81f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,351 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,351 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,356 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,356 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,356 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,357 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,357 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,357 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,362 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,362 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/4301457a68264bf6984f4a34103d85ba 2023-04-07 09:55:48,462 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,466 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,469 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,469 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,469 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,469 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,470 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,471 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@145ce4a0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,473 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,473 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,473 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,478 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,481 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,482 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,484 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,487 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,488 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,588 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,591 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,592 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,593 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,593 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,593 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,594 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,594 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,595 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,597 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30078b51[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,599 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,600 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,600 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,601 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,601 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,604 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,604 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,606 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,610 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,611 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,611 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861408621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861408622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861408622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861408623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861408623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,711 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,717 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,718 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,722 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,723 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,724 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60399a82[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,726 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,728 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,736 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,737 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,737 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,738 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,738 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,738 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,739 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,742 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,744 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,744 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,745 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,745 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/024012fd1324477ba28734625e982f39 2023-04-07 09:55:48,845 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,849 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,849 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,850 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,850 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,851 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,852 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,852 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,852 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,852 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,855 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7447ec9a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,861 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,861 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,863 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,864 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,867 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,868 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,868 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,868 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,868 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,968 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:48,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:48,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:48,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:48,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:48,971 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:48,971 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,971 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:48,971 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:48,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,973 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,973 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:48,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:48,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:48,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:48,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:48,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:48,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:48,974 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,975 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:48,975 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:48,975 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,976 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:48,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cd6f56[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:48,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:48,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:48,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:48,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:48,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:48,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:48,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:48,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:48,984 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:48,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:48,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:48,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:48,984 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:48,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:48,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:48,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:48,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,989 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:48,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:48,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:48,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:48,990 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:48,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:48,990 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:48,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:48,990 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:48,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:49,090 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:49,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:49,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:49,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:49,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:49,093 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:49,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:49,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:49,093 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:49,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,094 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,095 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:49,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:49,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,095 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:49,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:49,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:49,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:49,097 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,097 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:49,097 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:49,097 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:49,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,099 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:49,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1c31fd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:49,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:49,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,102 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:49,102 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,102 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:49,103 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:49,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:49,106 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:49,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:49,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:49,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:49,106 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:49,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,118 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,118 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,118 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:49,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:49,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:49,118 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:49,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:49,118 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:49,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:49,119 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:49,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,119 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:49,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:49,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861409127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:49,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861409129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:49,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861409129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:49,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861409130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:49,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861409131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04 2023-04-07 09:55:49,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/4301457a68264bf6984f4a34103d85ba as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4301457a68264bf6984f4a34103d85ba 2023-04-07 09:55:49,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4301457a68264bf6984f4a34103d85ba, entries=200, sequenceid=421, filesize=14.3 K 2023-04-07 09:55:49,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/024012fd1324477ba28734625e982f39 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/024012fd1324477ba28734625e982f39 2023-04-07 09:55:49,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/024012fd1324477ba28734625e982f39, entries=150, sequenceid=421, filesize=11.9 K 2023-04-07 09:55:49,219 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:49,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:49,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:49,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:49,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:49,222 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:49,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:49,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:49,222 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:49,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04 2023-04-07 09:55:49,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,225 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:49,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:49,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,232 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,232 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:49,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:49,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04, entries=150, sequenceid=421, filesize=11.9 K 2023-04-07 09:55:49,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:49,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:49,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:49,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:49,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:49,235 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c05568a99c8bbb1b0521e425f41a54f2 in 1277ms, sequenceid=421, compaction requested=true 2023-04-07 09:55:49,235 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:49,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:49,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:49,235 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:49,235 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:49,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-07 09:55:49,236 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:49,236 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:49,236 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:49,236 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,236 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:49,236 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/675384180f944d6e947d1d1c6689f733, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/dceafe5486794c5db274b3dc2ac88512, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4301457a68264bf6984f4a34103d85ba] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=38.6 K 2023-04-07 09:55:49,237 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,237 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/bbbe3f036c604ea88687442ac5bb32cd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c4cb2fbe0d154aa1a757397f0334af99, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/024012fd1324477ba28734625e982f39] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.2 K 2023-04-07 09:55:49,237 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 675384180f944d6e947d1d1c6689f733, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680861346954 2023-04-07 09:55:49,237 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting bbbe3f036c604ea88687442ac5bb32cd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680861346954 2023-04-07 09:55:49,237 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting dceafe5486794c5db274b3dc2ac88512, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680861347000 2023-04-07 09:55:49,237 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c4cb2fbe0d154aa1a757397f0334af99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680861347000 2023-04-07 09:55:49,238 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 4301457a68264bf6984f4a34103d85ba, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680861347641 2023-04-07 09:55:49,238 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 024012fd1324477ba28734625e982f39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680861347642 2023-04-07 09:55:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:49,244 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,244 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:49,244 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:55:49,244 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:49,261 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#539 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:49,277 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:49,707 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/9d5bac65afcb4adea5b36345437dda85 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/9d5bac65afcb4adea5b36345437dda85 2023-04-07 09:55:49,711 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 9d5bac65afcb4adea5b36345437dda85(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:49,711 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:49,711 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861349235; duration=0sec 2023-04-07 09:55:49,711 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:49,711 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:49,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:49,712 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:49,712 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,712 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d45acbd8456b40cfa09dbad872cb4b72, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b3139898f7564b949c3ea5953e26b59e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.2 K 2023-04-07 09:55:49,713 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting d45acbd8456b40cfa09dbad872cb4b72, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680861346954 2023-04-07 09:55:49,713 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b3139898f7564b949c3ea5953e26b59e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680861347000 2023-04-07 09:55:49,713 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 1c3b9827a4fd4d6c8e2ae6a2cae72e04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680861347642 2023-04-07 09:55:49,715 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/4636628405b2455682e05f89e3e33a47 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/4636628405b2455682e05f89e3e33a47 2023-04-07 09:55:49,718 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into 4636628405b2455682e05f89e3e33a47(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:49,718 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:49,718 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861349235; duration=0sec 2023-04-07 09:55:49,718 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#542 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:49,718 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:49,722 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/8e3a8d1cf20e4a3c83665151d77a2b21 2023-04-07 09:55:49,728 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/6ab8e51d5c5044dcbd42d5b9726959e9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/6ab8e51d5c5044dcbd42d5b9726959e9 2023-04-07 09:55:49,732 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 6ab8e51d5c5044dcbd42d5b9726959e9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:49,732 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:49,732 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861349235; duration=0sec 2023-04-07 09:55:49,732 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:49,736 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/1a6d3a4dbbe84eb095ec87af4c42be74 2023-04-07 09:55:49,743 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/7aec0b8e65bf46919eec66e758e595f8 2023-04-07 09:55:49,746 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/8e3a8d1cf20e4a3c83665151d77a2b21 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8e3a8d1cf20e4a3c83665151d77a2b21 2023-04-07 09:55:49,750 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8e3a8d1cf20e4a3c83665151d77a2b21, entries=150, sequenceid=442, filesize=11.9 K 2023-04-07 09:55:49,751 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/1a6d3a4dbbe84eb095ec87af4c42be74 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/1a6d3a4dbbe84eb095ec87af4c42be74 2023-04-07 09:55:49,754 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/1a6d3a4dbbe84eb095ec87af4c42be74, entries=150, sequenceid=442, filesize=11.9 K 2023-04-07 09:55:49,754 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/7aec0b8e65bf46919eec66e758e595f8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7aec0b8e65bf46919eec66e758e595f8 2023-04-07 09:55:49,757 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7aec0b8e65bf46919eec66e758e595f8, entries=150, sequenceid=442, filesize=11.9 K 2023-04-07 09:55:49,758 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for c05568a99c8bbb1b0521e425f41a54f2 in 514ms, sequenceid=442, compaction requested=false 2023-04-07 09:55:49,758 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:49,758 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:49,760 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,760 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27dd4796[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:49,762 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:49,762 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,764 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:49,764 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,765 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:49,766 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:49,770 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:49,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:49,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:49,770 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:49,770 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:49,771 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,771 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,771 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:49,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,772 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:49,778 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:49,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:49,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:49,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:49,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:49,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:49,778 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:49,779 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:49,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,779 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:49,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,879 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:49,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:49,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:49,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:49,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:49,884 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:49,884 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:49,884 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:49,884 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:49,884 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,884 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,886 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:49,886 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:49,887 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,887 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:49,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:49,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:49,888 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,888 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:49,888 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:49,888 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:49,890 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c354ccb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:49,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:49,901 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,902 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:49,902 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:49,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,902 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:49,903 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:49,906 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:49,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:49,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:49,906 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:49,906 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:49,908 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,908 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:49,908 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:49,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:49,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:49,910 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:49,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:49,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:49,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:49,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:49,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:49,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:49,912 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:49,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:49,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:49,912 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:49,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:49,913 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:49,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,013 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:50,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:50,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:50,016 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:50,016 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,016 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:50,016 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:50,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,017 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,018 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,018 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:50,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:50,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:50,019 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:50,020 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:50,020 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:50,020 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:50,021 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee29c4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:50,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:50,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:50,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:50,027 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:50,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:50,046 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:50,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:50,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:50,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:50,046 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:50,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,055 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:50,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:50,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,056 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:50,056 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:50,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:50,142 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:50,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:50,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:50,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:50,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:50,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:50,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:50,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/8bde76bfc68848d9b4bb8bb96eea5211 2023-04-07 09:55:50,156 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:50,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:50,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/0a5432c1713f4e8b9f8de257e86f4ab2 2023-04-07 09:55:50,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:50,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:50,160 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:50,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:50,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:50,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:50,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:50,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:50,165 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,165 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:50,165 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:50,165 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:50,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1763fd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:50,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:50,171 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861410163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680861350152, max timestamp=1680861350171Num uniques 859; 2023-04-07 09:55:50,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:50,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680861350152, max timestamp=1680861350171Num uniques 859; 2023-04-07 09:55:50,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:50,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680861350152, max timestamp=1680861350171Num uniques 859; 2023-04-07 09:55:50,174 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:55:50,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,174 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:50,174 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:50,174 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:50,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:50,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:50,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:50,178 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:50,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:50,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:50,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:50,178 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:50,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,181 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,183 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:50,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:50,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,185 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,187 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:50,187 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:50,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861410204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861410204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861410205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861410208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861410273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,287 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:50,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:50,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:50,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:50,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:50,290 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:50,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:50,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:50,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,290 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,291 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,291 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,292 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,292 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:50,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:50,293 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,293 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:50,293 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:50,293 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:50,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:50,294 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27bb78de[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:50,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:50,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,297 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:50,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:50,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,300 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:50,303 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:50,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:50,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:50,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:50,303 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:50,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:50,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,309 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:50,309 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:50,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861410309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861410313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861410313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861410319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,409 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:50,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:50,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:50,412 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:50,412 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,412 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:50,412 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:50,412 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,412 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,414 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:50,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:50,415 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:50,415 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,415 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:50,415 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:50,415 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:50,418 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@612e3bf6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:50,420 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:50,420 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,422 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:50,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,422 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:50,423 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,423 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:50,425 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:50,425 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:50,426 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:50,426 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:50,426 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:50,427 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,427 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,427 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,428 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:50,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:50,432 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:50,432 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:50,432 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:50,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861410479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861410516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861410518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861410518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861410526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,532 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:50,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:50,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:50,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:50,535 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:50,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:50,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:50,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,536 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,541 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,541 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,541 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,541 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:50,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:50,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:50,542 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,542 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:50,542 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:50,542 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:50,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:50,544 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,544 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:50,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ff8e1e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:50,546 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:50,546 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,549 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,549 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:50,550 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:50,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,551 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:50,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:50,554 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:50,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:50,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:50,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:50,554 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:50,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:50,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,556 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,557 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,559 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,559 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:50,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:50,559 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,559 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:50,559 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:50,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,559 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,559 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,560 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:50,560 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,560 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:50,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:50,560 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:50,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/8471f160da924ac99d64d2b4f2938370 2023-04-07 09:55:50,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/8bde76bfc68848d9b4bb8bb96eea5211 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8bde76bfc68848d9b4bb8bb96eea5211 2023-04-07 09:55:50,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8bde76bfc68848d9b4bb8bb96eea5211, entries=300, sequenceid=459, filesize=19.0 K 2023-04-07 09:55:50,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/0a5432c1713f4e8b9f8de257e86f4ab2 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0a5432c1713f4e8b9f8de257e86f4ab2 2023-04-07 09:55:50,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0a5432c1713f4e8b9f8de257e86f4ab2, entries=150, sequenceid=459, filesize=11.9 K 2023-04-07 09:55:50,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/8471f160da924ac99d64d2b4f2938370 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8471f160da924ac99d64d2b4f2938370 2023-04-07 09:55:50,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8471f160da924ac99d64d2b4f2938370, entries=150, sequenceid=459, filesize=11.9 K 2023-04-07 09:55:50,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for c05568a99c8bbb1b0521e425f41a54f2 in 441ms, sequenceid=459, compaction requested=true 2023-04-07 09:55:50,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:50,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:50,583 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:50,583 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:50,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:50,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:50,584 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44492 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:50,584 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:50,584 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:50,584 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,584 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:50,584 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/9d5bac65afcb4adea5b36345437dda85, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8e3a8d1cf20e4a3c83665151d77a2b21, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8bde76bfc68848d9b4bb8bb96eea5211] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=43.4 K 2023-04-07 09:55:50,584 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,584 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/4636628405b2455682e05f89e3e33a47, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/1a6d3a4dbbe84eb095ec87af4c42be74, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0a5432c1713f4e8b9f8de257e86f4ab2] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.3 K 2023-04-07 09:55:50,585 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 9d5bac65afcb4adea5b36345437dda85, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680861347642 2023-04-07 09:55:50,585 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4636628405b2455682e05f89e3e33a47, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680861347642 2023-04-07 09:55:50,585 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8e3a8d1cf20e4a3c83665151d77a2b21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680861347992 2023-04-07 09:55:50,585 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1a6d3a4dbbe84eb095ec87af4c42be74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680861347992 2023-04-07 09:55:50,585 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8bde76bfc68848d9b4bb8bb96eea5211, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680861350136 2023-04-07 09:55:50,585 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0a5432c1713f4e8b9f8de257e86f4ab2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680861350141 2023-04-07 09:55:50,591 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#548 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:50,593 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#549 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:50,609 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/f58dc881e1d64abfbcbf25aac009d2ec as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/f58dc881e1d64abfbcbf25aac009d2ec 2023-04-07 09:55:50,609 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/adfe6cec781545e79caffd9a46061971 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/adfe6cec781545e79caffd9a46061971 2023-04-07 09:55:50,613 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into adfe6cec781545e79caffd9a46061971(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:50,613 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:50,613 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861350583; duration=0sec 2023-04-07 09:55:50,613 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into f58dc881e1d64abfbcbf25aac009d2ec(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:50,613 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:50,613 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:50,613 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861350583; duration=0sec 2023-04-07 09:55:50,613 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:50,613 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:50,614 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:50,614 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:50,614 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,614 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/6ab8e51d5c5044dcbd42d5b9726959e9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7aec0b8e65bf46919eec66e758e595f8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8471f160da924ac99d64d2b4f2938370] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.3 K 2023-04-07 09:55:50,615 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 6ab8e51d5c5044dcbd42d5b9726959e9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680861347642 2023-04-07 09:55:50,615 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7aec0b8e65bf46919eec66e758e595f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680861347992 2023-04-07 09:55:50,615 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8471f160da924ac99d64d2b4f2938370, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680861350141 2023-04-07 09:55:50,619 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#550 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:50,660 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:50,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:50,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:50,663 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:50,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:50,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:50,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,663 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,666 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,666 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:50,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:50,666 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,666 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:50,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:50,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:50,667 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:50,668 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:50,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:50,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:50,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861410823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861410823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861410828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861410830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861410832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861410933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:50,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:50,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861410935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,026 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/ca66f2f074514410bb1a717e79328b23 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ca66f2f074514410bb1a717e79328b23 2023-04-07 09:55:51,030 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into ca66f2f074514410bb1a717e79328b23(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:51,030 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:51,030 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861350583; duration=0sec 2023-04-07 09:55:51,030 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:51,079 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/bf7f6ea069f0471d9c5657a01adac6f0 2023-04-07 09:55:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861411138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:51,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861411142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:51,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861411332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:51,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861411334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:51,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861411341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861411441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:51,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861411448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,488 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/33c50f1d674f4c0ca937514aee478d9d 2023-04-07 09:55:51,897 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/156a39f76fa944ea84de49c9e4599b30 2023-04-07 09:55:51,901 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/bf7f6ea069f0471d9c5657a01adac6f0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bf7f6ea069f0471d9c5657a01adac6f0 2023-04-07 09:55:51,906 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bf7f6ea069f0471d9c5657a01adac6f0, entries=150, sequenceid=494, filesize=11.9 K 2023-04-07 09:55:51,906 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/33c50f1d674f4c0ca937514aee478d9d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/33c50f1d674f4c0ca937514aee478d9d 2023-04-07 09:55:51,910 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/33c50f1d674f4c0ca937514aee478d9d, entries=150, sequenceid=494, filesize=11.9 K 2023-04-07 09:55:51,910 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/156a39f76fa944ea84de49c9e4599b30 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/156a39f76fa944ea84de49c9e4599b30 2023-04-07 09:55:51,913 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/156a39f76fa944ea84de49c9e4599b30, entries=150, sequenceid=494, filesize=11.9 K 2023-04-07 09:55:51,914 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for c05568a99c8bbb1b0521e425f41a54f2 in 1246ms, sequenceid=494, compaction requested=false 2023-04-07 09:55:51,914 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:51,914 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:51,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,916 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:51,916 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:51,916 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:51,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41261f77[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:51,917 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:51,917 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:51,919 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,919 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:51,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:51,922 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:51,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:51,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:51,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:51,923 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:51,927 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:51,927 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:51,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:51,927 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:51,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:51,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:51,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,930 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:51,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:51,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:51,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:51,931 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:51,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:51,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:51,931 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:51,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:51,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:51,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-07 09:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:51,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/b331e7120b564b2b86ef39e32a4b832a 2023-04-07 09:55:51,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/e4f01faa56104cbda795f059287f8b66 2023-04-07 09:55:51,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/93541b2a7b8e4f09993b63ef6420c244 2023-04-07 09:55:51,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/b331e7120b564b2b86ef39e32a4b832a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b331e7120b564b2b86ef39e32a4b832a 2023-04-07 09:55:51,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b331e7120b564b2b86ef39e32a4b832a, entries=200, sequenceid=508, filesize=14.3 K 2023-04-07 09:55:51,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/e4f01faa56104cbda795f059287f8b66 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e4f01faa56104cbda795f059287f8b66 2023-04-07 09:55:51,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e4f01faa56104cbda795f059287f8b66, entries=150, sequenceid=508, filesize=11.9 K 2023-04-07 09:55:51,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/93541b2a7b8e4f09993b63ef6420c244 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/93541b2a7b8e4f09993b63ef6420c244 2023-04-07 09:55:51,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/93541b2a7b8e4f09993b63ef6420c244, entries=150, sequenceid=508, filesize=11.9 K 2023-04-07 09:55:51,998 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c05568a99c8bbb1b0521e425f41a54f2 in 48ms, sequenceid=508, compaction requested=true 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:51,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:51,999 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:51,999 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:51,999 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:51,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:52,000 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:52,000 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:52,000 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:52,000 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,000 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:52,000 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/f58dc881e1d64abfbcbf25aac009d2ec, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bf7f6ea069f0471d9c5657a01adac6f0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b331e7120b564b2b86ef39e32a4b832a] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=38.8 K 2023-04-07 09:55:52,000 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,000 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/adfe6cec781545e79caffd9a46061971, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/33c50f1d674f4c0ca937514aee478d9d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e4f01faa56104cbda795f059287f8b66] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.4 K 2023-04-07 09:55:52,000 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting f58dc881e1d64abfbcbf25aac009d2ec, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680861350141 2023-04-07 09:55:52,001 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting adfe6cec781545e79caffd9a46061971, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680861350141 2023-04-07 09:55:52,002 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bf7f6ea069f0471d9c5657a01adac6f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680861350196 2023-04-07 09:55:52,002 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 33c50f1d674f4c0ca937514aee478d9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680861350196 2023-04-07 09:55:52,002 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting e4f01faa56104cbda795f059287f8b66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861350818 2023-04-07 09:55:52,002 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b331e7120b564b2b86ef39e32a4b832a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861350818 2023-04-07 09:55:52,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/85d79603d9d245f7a5cace608fd86050 2023-04-07 09:55:52,008 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#558 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:52,011 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#559 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:52,019 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/de47a2f49a284f059a4b56010cf55daf as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/de47a2f49a284f059a4b56010cf55daf 2023-04-07 09:55:52,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/271483c467234cc7abbc0eeb50df4e2c 2023-04-07 09:55:52,025 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/073b4a6e5351477db49c6eb485bad848 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/073b4a6e5351477db49c6eb485bad848 2023-04-07 09:55:52,025 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into de47a2f49a284f059a4b56010cf55daf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:52,025 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,026 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861351999; duration=0sec 2023-04-07 09:55:52,026 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:52,026 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:52,027 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:52,027 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:52,027 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,027 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ca66f2f074514410bb1a717e79328b23, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/156a39f76fa944ea84de49c9e4599b30, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/93541b2a7b8e4f09993b63ef6420c244] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.4 K 2023-04-07 09:55:52,028 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ca66f2f074514410bb1a717e79328b23, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680861350141 2023-04-07 09:55:52,029 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 156a39f76fa944ea84de49c9e4599b30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680861350196 2023-04-07 09:55:52,029 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 93541b2a7b8e4f09993b63ef6420c244, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861350818 2023-04-07 09:55:52,030 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 073b4a6e5351477db49c6eb485bad848(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:52,030 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,030 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861351999; duration=0sec 2023-04-07 09:55:52,030 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:52,031 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,036 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,038 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,039 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,039 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:52,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,040 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,040 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,040 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,040 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2424a88914384be7a28165294ad83a6d 2023-04-07 09:55:52,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,042 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#562 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/85d79603d9d245f7a5cace608fd86050 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/85d79603d9d245f7a5cace608fd86050 2023-04-07 09:55:52,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861412037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,047 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/85d79603d9d245f7a5cace608fd86050, entries=250, sequenceid=528, filesize=16.7 K 2023-04-07 09:55:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861412039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/271483c467234cc7abbc0eeb50df4e2c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/271483c467234cc7abbc0eeb50df4e2c 2023-04-07 09:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da7e205[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/271483c467234cc7abbc0eeb50df4e2c, entries=150, sequenceid=528, filesize=11.9 K 2023-04-07 09:55:52,054 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/2424a88914384be7a28165294ad83a6d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2424a88914384be7a28165294ad83a6d 2023-04-07 09:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,057 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/48d9c26201c04305ad5a4cab2a5aa89e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/48d9c26201c04305ad5a4cab2a5aa89e 2023-04-07 09:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,059 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,059 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2424a88914384be7a28165294ad83a6d, entries=150, sequenceid=528, filesize=11.9 K 2023-04-07 09:55:52,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for c05568a99c8bbb1b0521e425f41a54f2 in 62ms, sequenceid=528, compaction requested=false 2023-04-07 09:55:52,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,061 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,062 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 48d9c26201c04305ad5a4cab2a5aa89e(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:52,062 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,062 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861351999; duration=0sec 2023-04-07 09:55:52,062 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,063 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,066 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,066 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,066 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,067 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,067 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,067 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:52,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-07 09:55:52,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:52,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:52,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:52,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:52,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:52,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:52,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/c75b38bafec146c1bf808cf3766a2d80 2023-04-07 09:55:52,167 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/9646b7c4f5c3430eb1e0aa669f3f9e84 2023-04-07 09:55:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,170 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,170 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,173 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/60673f91a3164d4f94f1148407c5d48d 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,175 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,175 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,175 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,175 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/c75b38bafec146c1bf808cf3766a2d80 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c75b38bafec146c1bf808cf3766a2d80 2023-04-07 09:55:52,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c75b38bafec146c1bf808cf3766a2d80, entries=200, sequenceid=545, filesize=14.3 K 2023-04-07 09:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/9646b7c4f5c3430eb1e0aa669f3f9e84 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9646b7c4f5c3430eb1e0aa669f3f9e84 2023-04-07 09:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dc68fcc[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,183 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9646b7c4f5c3430eb1e0aa669f3f9e84, entries=150, sequenceid=545, filesize=11.9 K 2023-04-07 09:55:52,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/60673f91a3164d4f94f1148407c5d48d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/60673f91a3164d4f94f1148407c5d48d 2023-04-07 09:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/60673f91a3164d4f94f1148407c5d48d, entries=150, sequenceid=545, filesize=11.9 K 2023-04-07 09:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=40.25 KB/41220 for c05568a99c8bbb1b0521e425f41a54f2 in 40ms, sequenceid=545, compaction requested=true 2023-04-07 09:55:52,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:52,190 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:52,190 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:52,190 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,190 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,191 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:52,191 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:52,191 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:52,192 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:52,192 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,192 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,192 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/073b4a6e5351477db49c6eb485bad848, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/85d79603d9d245f7a5cace608fd86050, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c75b38bafec146c1bf808cf3766a2d80] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=43.6 K 2023-04-07 09:55:52,192 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/de47a2f49a284f059a4b56010cf55daf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/271483c467234cc7abbc0eeb50df4e2c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9646b7c4f5c3430eb1e0aa669f3f9e84] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.5 K 2023-04-07 09:55:52,192 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 073b4a6e5351477db49c6eb485bad848, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861350818 2023-04-07 09:55:52,192 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting de47a2f49a284f059a4b56010cf55daf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861350818 2023-04-07 09:55:52,192 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 85d79603d9d245f7a5cace608fd86050, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1680861351978 2023-04-07 09:55:52,192 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 271483c467234cc7abbc0eeb50df4e2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1680861351978 2023-04-07 09:55:52,193 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting c75b38bafec146c1bf808cf3766a2d80, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680861352006 2023-04-07 09:55:52,193 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9646b7c4f5c3430eb1e0aa669f3f9e84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680861352006 2023-04-07 09:55:52,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,194 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:52,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:52,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:52,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:52,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:52,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:52,197 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,199 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,201 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#567 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:52,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,202 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,202 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,202 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,203 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,217 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/fe4205cd25694278a533b7f21947c7db as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/fe4205cd25694278a533b7f21947c7db 2023-04-07 09:55:52,220 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into fe4205cd25694278a533b7f21947c7db(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:52,220 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,220 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=13, startTime=1680861352190; duration=0sec 2023-04-07 09:55:52,220 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:52,221 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:55:52,221 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:55:52,221 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:52,221 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,221 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/48d9c26201c04305ad5a4cab2a5aa89e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2424a88914384be7a28165294ad83a6d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/60673f91a3164d4f94f1148407c5d48d] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=36.5 K 2023-04-07 09:55:52,222 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 48d9c26201c04305ad5a4cab2a5aa89e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680861350818 2023-04-07 09:55:52,222 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 2424a88914384be7a28165294ad83a6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1680861351978 2023-04-07 09:55:52,222 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 60673f91a3164d4f94f1148407c5d48d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680861352006 2023-04-07 09:55:52,226 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#569 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:52,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861352225, max timestamp=1680861352261Num uniques 859; 2023-04-07 09:55:52,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:52,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861352225, max timestamp=1680861352261Num uniques 859; 2023-04-07 09:55:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861412254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861352225, max timestamp=1680861352261Num uniques 859; 2023-04-07 09:55:52,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:52,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:52,303 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,306 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,307 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,307 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:52,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,314 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,314 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,314 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41257b14[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,324 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,324 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,325 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,325 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,326 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,326 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,327 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,330 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,331 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,331 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1680861412352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861412356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861412357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861412361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861412364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,431 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,435 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,435 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,438 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,438 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,438 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:52,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,439 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,440 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,440 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,440 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,441 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,442 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@308909e0[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,448 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,448 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,449 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,454 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:52,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,458 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,458 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,458 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861412465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861412468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861412469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,558 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,563 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,566 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861412570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,576 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,576 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,579 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32bc723e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,582 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,586 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,586 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,586 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,587 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,587 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,591 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,595 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,595 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,595 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,599 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,599 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,599 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,601 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,604 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:52,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,605 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,605 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/3cb7f0a79cb4407cbb031abae4d296fb 2023-04-07 09:55:52,618 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/6ecc614c4bbe492b9bc9e89c97f8df68 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6ecc614c4bbe492b9bc9e89c97f8df68 2023-04-07 09:55:52,624 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into 6ecc614c4bbe492b9bc9e89c97f8df68(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:52,624 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,625 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=13, startTime=1680861352190; duration=0sec 2023-04-07 09:55:52,625 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:52,637 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/81f667710fdb47b29c57d7f751ce9450 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81f667710fdb47b29c57d7f751ce9450 2023-04-07 09:55:52,644 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 81f667710fdb47b29c57d7f751ce9450(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:52,644 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:52,644 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=13, startTime=1680861352190; duration=0sec 2023-04-07 09:55:52,644 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:52,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861412673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861412676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861412677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,705 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,711 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,713 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,714 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,715 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,715 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,715 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,715 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,717 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b1d334a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,721 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,721 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,722 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,722 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,724 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,724 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,727 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,727 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,727 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,731 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:52,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,731 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,732 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,732 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,732 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,833 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,836 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,838 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,838 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,840 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,840 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,842 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@418a00e6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,845 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,847 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,847 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,848 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,849 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,849 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,849 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,849 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,852 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,852 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,852 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,854 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,854 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,855 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,858 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,858 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,859 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:52,858 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861412878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,959 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:52,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:52,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:52,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:52,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:52,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,962 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,967 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,967 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:52,969 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,969 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:52,969 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:52,969 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:52,971 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3934d73e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:52,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:52,972 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,974 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:52,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:52,974 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:52,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,974 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:52,975 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:52,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:52,978 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:52,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:52,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:52,978 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:52,978 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:52,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861412975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,983 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:52,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,983 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:52,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,984 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:52,986 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:52,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:52,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:52,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:52,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:52,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:52,987 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:52,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:52,987 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:52,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:52,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:52,987 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:52,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861412988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:52,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:52,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861412989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,052 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/86cd02a5122c4d96a06d7e6c775ca562 2023-04-07 09:55:53,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/989fe31637d44eb5880c647fdf016928 2023-04-07 09:55:53,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/3cb7f0a79cb4407cbb031abae4d296fb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3cb7f0a79cb4407cbb031abae4d296fb 2023-04-07 09:55:53,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3cb7f0a79cb4407cbb031abae4d296fb, entries=250, sequenceid=556, filesize=16.7 K 2023-04-07 09:55:53,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/86cd02a5122c4d96a06d7e6c775ca562 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/86cd02a5122c4d96a06d7e6c775ca562 2023-04-07 09:55:53,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/86cd02a5122c4d96a06d7e6c775ca562, entries=150, sequenceid=556, filesize=11.9 K 2023-04-07 09:55:53,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/989fe31637d44eb5880c647fdf016928 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/989fe31637d44eb5880c647fdf016928 2023-04-07 09:55:53,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/989fe31637d44eb5880c647fdf016928, entries=150, sequenceid=556, filesize=11.9 K 2023-04-07 09:55:53,078 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c05568a99c8bbb1b0521e425f41a54f2 in 881ms, sequenceid=556, compaction requested=false 2023-04-07 09:55:53,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:53,087 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:53,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:53,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:53,090 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:53,091 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:53,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:53,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:53,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:53,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:53,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:53,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:53,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:53,093 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:53,093 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-07 09:55:53,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:53,094 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:53,094 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:53,094 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:53,094 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:53,094 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:53,390 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:53,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861413421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:53,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49496 deadline: 1680861413481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:53,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49482 deadline: 1680861413495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,500 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/cd57fd5ecf4742f58bb31fce33b9d8cc 2023-04-07 09:55:53,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:53,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49492 deadline: 1680861413498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,509 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/a350e14933aa45f4a20ae9569593e7cc 2023-04-07 09:55:53,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:53,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861413525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:55:53,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49554 deadline: 1680861413728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,917 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/870445cb74754f6db245034f5a09c6ce 2023-04-07 09:55:53,921 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/cd57fd5ecf4742f58bb31fce33b9d8cc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/cd57fd5ecf4742f58bb31fce33b9d8cc 2023-04-07 09:55:53,924 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/cd57fd5ecf4742f58bb31fce33b9d8cc, entries=150, sequenceid=595, filesize=11.9 K 2023-04-07 09:55:53,924 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/a350e14933aa45f4a20ae9569593e7cc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a350e14933aa45f4a20ae9569593e7cc 2023-04-07 09:55:53,927 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a350e14933aa45f4a20ae9569593e7cc, entries=150, sequenceid=595, filesize=11.9 K 2023-04-07 09:55:53,928 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/870445cb74754f6db245034f5a09c6ce as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/870445cb74754f6db245034f5a09c6ce 2023-04-07 09:55:53,931 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/870445cb74754f6db245034f5a09c6ce, entries=150, sequenceid=595, filesize=11.9 K 2023-04-07 09:55:53,931 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for c05568a99c8bbb1b0521e425f41a54f2 in 838ms, sequenceid=595, compaction requested=true 2023-04-07 09:55:53,931 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:53,931 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:53,934 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68486fa4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:53,936 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:53,936 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,937 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:53,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,937 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:53,939 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:53,941 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:53,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:53,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:53,941 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:53,941 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:53,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:53,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:53,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:53,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:53,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:53,948 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:53,948 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:53,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:53,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:53,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:53,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:53,949 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:53,949 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:54,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:54,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:54,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:54,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/385f1416b11647a1908902cc54c1d9ba 2023-04-07 09:55:54,049 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:54,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:54,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:54,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:54,052 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:54,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:54,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:54,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:54,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,054 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:54,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:54,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,054 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:54,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:54,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:54,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/42434b90131d4c528dde25620d01c2df 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:54,056 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,056 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:54,056 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:54,056 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a8d9684[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:54,061 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:54,062 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,064 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,064 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:54,065 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/3feb6c7c530f4b03820ddcbdf265c02c 2023-04-07 09:55:54,065 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:54,065 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:54,065 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:54,068 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:54,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/385f1416b11647a1908902cc54c1d9ba as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/385f1416b11647a1908902cc54c1d9ba 2023-04-07 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:54,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:54,071 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:54,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:54,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:54,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:54,071 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:54,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:54,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/385f1416b11647a1908902cc54c1d9ba, entries=200, sequenceid=607, filesize=14.3 K 2023-04-07 09:55:54,073 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:54,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:54,074 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/42434b90131d4c528dde25620d01c2df as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/42434b90131d4c528dde25620d01c2df 2023-04-07 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,076 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/42434b90131d4c528dde25620d01c2df, entries=150, sequenceid=607, filesize=11.9 K 2023-04-07 09:55:54,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/3feb6c7c530f4b03820ddcbdf265c02c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/3feb6c7c530f4b03820ddcbdf265c02c 2023-04-07 09:55:54,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:54,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:54,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:54,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:54,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:54,079 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:54,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:54,079 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:54,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:54,079 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:54,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/3feb6c7c530f4b03820ddcbdf265c02c, entries=150, sequenceid=607, filesize=11.9 K 2023-04-07 09:55:54,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=40.25 KB/41220 for c05568a99c8bbb1b0521e425f41a54f2 in 44ms, sequenceid=607, compaction requested=true 2023-04-07 09:55:54,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:54,083 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:54,083 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:54,083 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:54,083 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:54,083 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:54,084 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:54,084 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:54,084 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/A is initiating minor compaction (all files) 2023-04-07 09:55:54,084 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/B is initiating minor compaction (all files) 2023-04-07 09:55:54,084 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/A in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,084 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/B in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,084 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6ecc614c4bbe492b9bc9e89c97f8df68, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3cb7f0a79cb4407cbb031abae4d296fb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/cd57fd5ecf4742f58bb31fce33b9d8cc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/385f1416b11647a1908902cc54c1d9ba] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=55.6 K 2023-04-07 09:55:54,084 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/fe4205cd25694278a533b7f21947c7db, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/86cd02a5122c4d96a06d7e6c775ca562, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a350e14933aa45f4a20ae9569593e7cc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/42434b90131d4c528dde25620d01c2df] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=48.5 K 2023-04-07 09:55:54,084 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 6ecc614c4bbe492b9bc9e89c97f8df68, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680861352006 2023-04-07 09:55:54,084 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fe4205cd25694278a533b7f21947c7db, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680861352006 2023-04-07 09:55:54,085 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3cb7f0a79cb4407cbb031abae4d296fb, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680861352168 2023-04-07 09:55:54,085 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 86cd02a5122c4d96a06d7e6c775ca562, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680861352168 2023-04-07 09:55:54,085 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting cd57fd5ecf4742f58bb31fce33b9d8cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680861352354 2023-04-07 09:55:54,085 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a350e14933aa45f4a20ae9569593e7cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680861352354 2023-04-07 09:55:54,085 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 385f1416b11647a1908902cc54c1d9ba, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1680861353410 2023-04-07 09:55:54,085 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 42434b90131d4c528dde25620d01c2df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1680861353410 2023-04-07 09:55:54,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:54,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:55:54,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:54,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:54,089 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:54,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:54,089 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:54,089 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:55:54,092 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#B#compaction#578 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:54,094 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#A#compaction#579 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:54,107 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/c7a365e0a75743ff841148f52cc3fbfd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c7a365e0a75743ff841148f52cc3fbfd 2023-04-07 09:55:54,111 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/A of c05568a99c8bbb1b0521e425f41a54f2 into c7a365e0a75743ff841148f52cc3fbfd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:54,111 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:54,111 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/A, priority=12, startTime=1680861354083; duration=0sec 2023-04-07 09:55:54,111 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:55:54,111 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:55:54,112 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:55:54,112 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): c05568a99c8bbb1b0521e425f41a54f2/C is initiating minor compaction (all files) 2023-04-07 09:55:54,112 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c05568a99c8bbb1b0521e425f41a54f2/C in TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,112 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81f667710fdb47b29c57d7f751ce9450, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/989fe31637d44eb5880c647fdf016928, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/870445cb74754f6db245034f5a09c6ce, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/3feb6c7c530f4b03820ddcbdf265c02c] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp, totalSize=48.5 K 2023-04-07 09:55:54,113 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 81f667710fdb47b29c57d7f751ce9450, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1680861352006 2023-04-07 09:55:54,113 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 989fe31637d44eb5880c647fdf016928, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680861352168 2023-04-07 09:55:54,113 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 870445cb74754f6db245034f5a09c6ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1680861352354 2023-04-07 09:55:54,114 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3feb6c7c530f4b03820ddcbdf265c02c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1680861353410 2023-04-07 09:55:54,119 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): c05568a99c8bbb1b0521e425f41a54f2#C#compaction#581 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:55:54,124 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/8ef51983a15d4e8694d0a613ea748585 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8ef51983a15d4e8694d0a613ea748585 2023-04-07 09:55:54,127 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/C of c05568a99c8bbb1b0521e425f41a54f2 into 8ef51983a15d4e8694d0a613ea748585(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:54,127 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:54,127 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/C, priority=12, startTime=1680861354083; duration=0sec 2023-04-07 09:55:54,127 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:54,179 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:54,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:54,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:55:54,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:55:54,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:55:54,182 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:55:54,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:54,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:55:54,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:55:54,183 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,183 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,184 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:54,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:54,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:55:54,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:55:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:55:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:55:54,186 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,186 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. started... 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:55:54,186 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. as already flushing 2023-04-07 09:55:54,186 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:55:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b913e03[Count = 0] remaining members to acquire global barrier 2023-04-07 09:55:54,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:55:54,190 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,191 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:55:54,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:55:54,191 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:55:54,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:55:54,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:55:54,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:54,193 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:55:54,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:55:54,195 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:55:54,195 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:55:54,195 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:55:54,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,197 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:54,197 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:55:54,199 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:55:54,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:55:54,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:55:54,205 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,205 DEBUG [Thread-3509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ac98310 to 127.0.0.1:49660 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:55:54,205 DEBUG [Thread-3505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:49660 2023-04-07 09:55:54,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:55:54,206 DEBUG [Thread-3505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:55:54,205 DEBUG [Thread-3509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,205 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:55:54,206 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:55:54,205 DEBUG [Thread-3500] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:55:54,206 DEBUG [Thread-3500] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:55:54,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:55:54,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861354173, max timestamp=1680861354206Num uniques 859; 2023-04-07 09:55:54,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861354173, max timestamp=1680861354206Num uniques 859; 2023-04-07 09:55:54,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:54,209 DEBUG [Thread-3494] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:49660 2023-04-07 09:55:54,209 DEBUG [Thread-3494] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:54,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861354173, max timestamp=1680861354206Num uniques 859; 2023-04-07 09:55:54,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:55:54,209 DEBUG [Thread-3507] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:49660 2023-04-07 09:55:54,209 DEBUG [Thread-3507] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,212 DEBUG [Thread-3501] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:49660 2023-04-07 09:55:54,213 DEBUG [Thread-3501] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,213 DEBUG [Thread-3503] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:49660 2023-04-07 09:55:54,213 DEBUG [Thread-3503] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,306 DEBUG [Thread-3500] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:55:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:55:54,374 DEBUG [Thread-3498] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:49660 2023-04-07 09:55:54,374 DEBUG [Thread-3498] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,494 DEBUG [Thread-3492] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:49660 2023-04-07 09:55:54,494 DEBUG [Thread-3492] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,502 DEBUG [Thread-3490] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:49660 2023-04-07 09:55:54,502 DEBUG [Thread-3490] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,504 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/6a23604a299b475d9638df3e86a36dda as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/6a23604a299b475d9638df3e86a36dda 2023-04-07 09:55:54,505 DEBUG [Thread-3496] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:49660 2023-04-07 09:55:54,505 DEBUG [Thread-3496] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,505 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 145 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 2103 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 6308 rows 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 2096 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 6287 rows 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 2112 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 6334 rows 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 2112 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 6334 rows 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(402): scanned 2107 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(403): verified 6320 rows 2023-04-07 09:55:54,506 INFO [Listener at localhost/39511] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 09:55:54,506 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:49660 2023-04-07 09:55:54,507 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:55:54,509 INFO [Listener at localhost/39511] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 09:55:54,509 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c05568a99c8bbb1b0521e425f41a54f2/B of c05568a99c8bbb1b0521e425f41a54f2 into 6a23604a299b475d9638df3e86a36dda(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:55:54,510 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:54,510 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2., storeName=c05568a99c8bbb1b0521e425f41a54f2/B, priority=12, startTime=1680861354083; duration=0sec 2023-04-07 09:55:54,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 09:55:54,510 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:55:54,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:54,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/d519c7017e4f4e82b6234d5a5fe8b34f 2023-04-07 09:55:54,514 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861354514"}]},"ts":"1680861354514"} 2023-04-07 09:55:54,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 09:55:54,516 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 09:55:54,518 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 09:55:54,518 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c05568a99c8bbb1b0521e425f41a54f2, UNASSIGN}] 2023-04-07 09:55:54,520 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c05568a99c8bbb1b0521e425f41a54f2, UNASSIGN 2023-04-07 09:55:54,521 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=c05568a99c8bbb1b0521e425f41a54f2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:54,521 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861354520"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861354520"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861354520"}]},"ts":"1680861354520"} 2023-04-07 09:55:54,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:54,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 09:55:54,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:54,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c05568a99c8bbb1b0521e425f41a54f2, disabling compactions & flushes 2023-04-07 09:55:54,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:54,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 09:55:54,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/ddcc667e676646678f1bbedfa33dd2fc 2023-04-07 09:55:55,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 09:55:55,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e2c83df02d954a16b6b8f27a4b52302f 2023-04-07 09:55:55,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/d519c7017e4f4e82b6234d5a5fe8b34f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d519c7017e4f4e82b6234d5a5fe8b34f 2023-04-07 09:55:55,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d519c7017e4f4e82b6234d5a5fe8b34f, entries=200, sequenceid=617, filesize=14.3 K 2023-04-07 09:55:55,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/ddcc667e676646678f1bbedfa33dd2fc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/ddcc667e676646678f1bbedfa33dd2fc 2023-04-07 09:55:55,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/ddcc667e676646678f1bbedfa33dd2fc, entries=150, sequenceid=617, filesize=11.9 K 2023-04-07 09:55:55,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/e2c83df02d954a16b6b8f27a4b52302f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e2c83df02d954a16b6b8f27a4b52302f 2023-04-07 09:55:55,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e2c83df02d954a16b6b8f27a4b52302f, entries=150, sequenceid=617, filesize=11.9 K 2023-04-07 09:55:55,349 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=53.67 KB/54960 for c05568a99c8bbb1b0521e425f41a54f2 in 1261ms, sequenceid=617, compaction requested=false 2023-04-07 09:55:55,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:55,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. after waiting 0 ms 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:55,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c05568a99c8bbb1b0521e425f41a54f2 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=A 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=B 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c05568a99c8bbb1b0521e425f41a54f2, store=C 2023-04-07 09:55:55,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:55:55,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 09:55:55,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/ca242b6be6b043389dc393ceada94634 2023-04-07 09:55:56,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/8b14f4fb3b3c400b97e7093a5e2a7ebe 2023-04-07 09:55:56,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/a60b4f381d3a4e1ba911ab684f127c53 2023-04-07 09:55:56,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/A/ca242b6be6b043389dc393ceada94634 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ca242b6be6b043389dc393ceada94634 2023-04-07 09:55:56,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ca242b6be6b043389dc393ceada94634, entries=150, sequenceid=645, filesize=11.9 K 2023-04-07 09:55:56,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/B/8b14f4fb3b3c400b97e7093a5e2a7ebe as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8b14f4fb3b3c400b97e7093a5e2a7ebe 2023-04-07 09:55:56,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8b14f4fb3b3c400b97e7093a5e2a7ebe, entries=150, sequenceid=645, filesize=11.9 K 2023-04-07 09:55:56,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/.tmp/C/a60b4f381d3a4e1ba911ab684f127c53 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/a60b4f381d3a4e1ba911ab684f127c53 2023-04-07 09:55:56,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/a60b4f381d3a4e1ba911ab684f127c53, entries=150, sequenceid=645, filesize=11.9 K 2023-04-07 09:55:56,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=0 B/0 for c05568a99c8bbb1b0521e425f41a54f2 in 1234ms, sequenceid=645, compaction requested=true 2023-04-07 09:55:56,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/e3b319390c7846e6aab03df04f570e0c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/a5ad922143e249f484608682651a8500, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc11397083a44bafb8188fb3f8a9b68a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/237ccaa45bca4b6098c2c4ae0e97e78f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d932944dea6d436682b81b11e41bc09a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b49e566e0b034ee9a4f5400c3b0399cb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/306a61aa697d45d58cb04813ffa139be, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/778b0b0f4d0d45fa9af183ed29e800a9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c585c5c35f804360a6f17fbad6444ab2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4f1085d336304924ad0fe851eab8b3a7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/887a4bbaf32845cdad766da0646f3873, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/99c1c6aa203b4481bdab87c114e9002e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/0589d37d82ed4f1b9e29d2cdbb4da5da, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ccd89302788845828143db0c82804c67, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/39c1ae15e49f424aac159b96a6f784ac, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/675384180f944d6e947d1d1c6689f733, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/dceafe5486794c5db274b3dc2ac88512, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4301457a68264bf6984f4a34103d85ba, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/9d5bac65afcb4adea5b36345437dda85, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8e3a8d1cf20e4a3c83665151d77a2b21, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8bde76bfc68848d9b4bb8bb96eea5211, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/f58dc881e1d64abfbcbf25aac009d2ec, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bf7f6ea069f0471d9c5657a01adac6f0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b331e7120b564b2b86ef39e32a4b832a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/073b4a6e5351477db49c6eb485bad848, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/85d79603d9d245f7a5cace608fd86050, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c75b38bafec146c1bf808cf3766a2d80, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6ecc614c4bbe492b9bc9e89c97f8df68, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3cb7f0a79cb4407cbb031abae4d296fb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/cd57fd5ecf4742f58bb31fce33b9d8cc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/385f1416b11647a1908902cc54c1d9ba] to archive 2023-04-07 09:55:56,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:56,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/e3b319390c7846e6aab03df04f570e0c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/e3b319390c7846e6aab03df04f570e0c 2023-04-07 09:55:56,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/a5ad922143e249f484608682651a8500 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/a5ad922143e249f484608682651a8500 2023-04-07 09:55:56,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc11397083a44bafb8188fb3f8a9b68a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bc11397083a44bafb8188fb3f8a9b68a 2023-04-07 09:55:56,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/237ccaa45bca4b6098c2c4ae0e97e78f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/237ccaa45bca4b6098c2c4ae0e97e78f 2023-04-07 09:55:56,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d932944dea6d436682b81b11e41bc09a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d932944dea6d436682b81b11e41bc09a 2023-04-07 09:55:56,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b49e566e0b034ee9a4f5400c3b0399cb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b49e566e0b034ee9a4f5400c3b0399cb 2023-04-07 09:55:56,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/306a61aa697d45d58cb04813ffa139be to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/306a61aa697d45d58cb04813ffa139be 2023-04-07 09:55:56,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/778b0b0f4d0d45fa9af183ed29e800a9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/778b0b0f4d0d45fa9af183ed29e800a9 2023-04-07 09:55:56,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c585c5c35f804360a6f17fbad6444ab2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c585c5c35f804360a6f17fbad6444ab2 2023-04-07 09:55:56,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4f1085d336304924ad0fe851eab8b3a7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4f1085d336304924ad0fe851eab8b3a7 2023-04-07 09:55:56,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/887a4bbaf32845cdad766da0646f3873 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/887a4bbaf32845cdad766da0646f3873 2023-04-07 09:55:56,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/99c1c6aa203b4481bdab87c114e9002e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/99c1c6aa203b4481bdab87c114e9002e 2023-04-07 09:55:56,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/0589d37d82ed4f1b9e29d2cdbb4da5da to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/0589d37d82ed4f1b9e29d2cdbb4da5da 2023-04-07 09:55:56,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ccd89302788845828143db0c82804c67 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ccd89302788845828143db0c82804c67 2023-04-07 09:55:56,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/39c1ae15e49f424aac159b96a6f784ac to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/39c1ae15e49f424aac159b96a6f784ac 2023-04-07 09:55:56,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/675384180f944d6e947d1d1c6689f733 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/675384180f944d6e947d1d1c6689f733 2023-04-07 09:55:56,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/dceafe5486794c5db274b3dc2ac88512 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/dceafe5486794c5db274b3dc2ac88512 2023-04-07 09:55:56,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4301457a68264bf6984f4a34103d85ba to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/4301457a68264bf6984f4a34103d85ba 2023-04-07 09:55:56,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/9d5bac65afcb4adea5b36345437dda85 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/9d5bac65afcb4adea5b36345437dda85 2023-04-07 09:55:56,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8e3a8d1cf20e4a3c83665151d77a2b21 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8e3a8d1cf20e4a3c83665151d77a2b21 2023-04-07 09:55:56,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8bde76bfc68848d9b4bb8bb96eea5211 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/8bde76bfc68848d9b4bb8bb96eea5211 2023-04-07 09:55:56,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/f58dc881e1d64abfbcbf25aac009d2ec to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/f58dc881e1d64abfbcbf25aac009d2ec 2023-04-07 09:55:56,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bf7f6ea069f0471d9c5657a01adac6f0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/bf7f6ea069f0471d9c5657a01adac6f0 2023-04-07 09:55:56,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b331e7120b564b2b86ef39e32a4b832a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/b331e7120b564b2b86ef39e32a4b832a 2023-04-07 09:55:56,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/073b4a6e5351477db49c6eb485bad848 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/073b4a6e5351477db49c6eb485bad848 2023-04-07 09:55:56,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/85d79603d9d245f7a5cace608fd86050 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/85d79603d9d245f7a5cace608fd86050 2023-04-07 09:55:56,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c75b38bafec146c1bf808cf3766a2d80 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c75b38bafec146c1bf808cf3766a2d80 2023-04-07 09:55:56,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6ecc614c4bbe492b9bc9e89c97f8df68 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/6ecc614c4bbe492b9bc9e89c97f8df68 2023-04-07 09:55:56,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3cb7f0a79cb4407cbb031abae4d296fb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/3cb7f0a79cb4407cbb031abae4d296fb 2023-04-07 09:55:56,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/cd57fd5ecf4742f58bb31fce33b9d8cc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/cd57fd5ecf4742f58bb31fce33b9d8cc 2023-04-07 09:55:56,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/385f1416b11647a1908902cc54c1d9ba to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/385f1416b11647a1908902cc54c1d9ba 2023-04-07 09:55:56,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/81863bf166544ae0b40a0dbc6000cb52, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0d8637e298b04ce1b5016d2180651f61, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a743115b00ac498e8dad284aae91a361, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/157230b72aa34964bb793ace3dc808ee, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/34c47edef79b4680a929bdc2d5a20517, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/2a5111cf93be4827a1f6b61fc72c4341, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f2719870af17455b86bb0f66ea4dfe44, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/43de347be9e6412aaa27be6c54358d5d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/62c57bbee2314385ad9203faec28d4cc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/06f3350e9dbd4a75889dd97c2f1969dc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/04856b55ec0541949b42cc60d70aa23b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c9c7d31353ba4235adfee6cddac9dd6b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/aa22b3ad7f494beaa047cfb9cb4b1653, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f572f5785dc8420fa6abfa0aaa21c939, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/bbbe3f036c604ea88687442ac5bb32cd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/553a5675dc7c4dd0a3db0206bd455a1f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c4cb2fbe0d154aa1a757397f0334af99, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/4636628405b2455682e05f89e3e33a47, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/024012fd1324477ba28734625e982f39, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/1a6d3a4dbbe84eb095ec87af4c42be74, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/adfe6cec781545e79caffd9a46061971, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0a5432c1713f4e8b9f8de257e86f4ab2, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/33c50f1d674f4c0ca937514aee478d9d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/de47a2f49a284f059a4b56010cf55daf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e4f01faa56104cbda795f059287f8b66, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/271483c467234cc7abbc0eeb50df4e2c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/fe4205cd25694278a533b7f21947c7db, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9646b7c4f5c3430eb1e0aa669f3f9e84, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/86cd02a5122c4d96a06d7e6c775ca562, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a350e14933aa45f4a20ae9569593e7cc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/42434b90131d4c528dde25620d01c2df] to archive 2023-04-07 09:55:56,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:56,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/81863bf166544ae0b40a0dbc6000cb52 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/81863bf166544ae0b40a0dbc6000cb52 2023-04-07 09:55:56,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0d8637e298b04ce1b5016d2180651f61 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0d8637e298b04ce1b5016d2180651f61 2023-04-07 09:55:56,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a743115b00ac498e8dad284aae91a361 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a743115b00ac498e8dad284aae91a361 2023-04-07 09:55:56,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/157230b72aa34964bb793ace3dc808ee to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/157230b72aa34964bb793ace3dc808ee 2023-04-07 09:55:56,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/34c47edef79b4680a929bdc2d5a20517 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/34c47edef79b4680a929bdc2d5a20517 2023-04-07 09:55:56,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/2a5111cf93be4827a1f6b61fc72c4341 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/2a5111cf93be4827a1f6b61fc72c4341 2023-04-07 09:55:56,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f2719870af17455b86bb0f66ea4dfe44 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f2719870af17455b86bb0f66ea4dfe44 2023-04-07 09:55:56,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/43de347be9e6412aaa27be6c54358d5d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/43de347be9e6412aaa27be6c54358d5d 2023-04-07 09:55:56,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 09:55:56,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/62c57bbee2314385ad9203faec28d4cc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/62c57bbee2314385ad9203faec28d4cc 2023-04-07 09:55:56,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/06f3350e9dbd4a75889dd97c2f1969dc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/06f3350e9dbd4a75889dd97c2f1969dc 2023-04-07 09:55:56,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/04856b55ec0541949b42cc60d70aa23b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/04856b55ec0541949b42cc60d70aa23b 2023-04-07 09:55:56,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c9c7d31353ba4235adfee6cddac9dd6b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c9c7d31353ba4235adfee6cddac9dd6b 2023-04-07 09:55:56,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/aa22b3ad7f494beaa047cfb9cb4b1653 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/aa22b3ad7f494beaa047cfb9cb4b1653 2023-04-07 09:55:56,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f572f5785dc8420fa6abfa0aaa21c939 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/f572f5785dc8420fa6abfa0aaa21c939 2023-04-07 09:55:56,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/bbbe3f036c604ea88687442ac5bb32cd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/bbbe3f036c604ea88687442ac5bb32cd 2023-04-07 09:55:56,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/553a5675dc7c4dd0a3db0206bd455a1f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/553a5675dc7c4dd0a3db0206bd455a1f 2023-04-07 09:55:56,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c4cb2fbe0d154aa1a757397f0334af99 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/c4cb2fbe0d154aa1a757397f0334af99 2023-04-07 09:55:56,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/4636628405b2455682e05f89e3e33a47 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/4636628405b2455682e05f89e3e33a47 2023-04-07 09:55:56,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/024012fd1324477ba28734625e982f39 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/024012fd1324477ba28734625e982f39 2023-04-07 09:55:56,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/1a6d3a4dbbe84eb095ec87af4c42be74 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/1a6d3a4dbbe84eb095ec87af4c42be74 2023-04-07 09:55:56,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/adfe6cec781545e79caffd9a46061971 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/adfe6cec781545e79caffd9a46061971 2023-04-07 09:55:56,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0a5432c1713f4e8b9f8de257e86f4ab2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/0a5432c1713f4e8b9f8de257e86f4ab2 2023-04-07 09:55:56,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/33c50f1d674f4c0ca937514aee478d9d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/33c50f1d674f4c0ca937514aee478d9d 2023-04-07 09:55:56,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/de47a2f49a284f059a4b56010cf55daf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/de47a2f49a284f059a4b56010cf55daf 2023-04-07 09:55:56,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e4f01faa56104cbda795f059287f8b66 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/e4f01faa56104cbda795f059287f8b66 2023-04-07 09:55:56,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/271483c467234cc7abbc0eeb50df4e2c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/271483c467234cc7abbc0eeb50df4e2c 2023-04-07 09:55:56,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/fe4205cd25694278a533b7f21947c7db to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/fe4205cd25694278a533b7f21947c7db 2023-04-07 09:55:56,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9646b7c4f5c3430eb1e0aa669f3f9e84 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/9646b7c4f5c3430eb1e0aa669f3f9e84 2023-04-07 09:55:56,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/86cd02a5122c4d96a06d7e6c775ca562 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/86cd02a5122c4d96a06d7e6c775ca562 2023-04-07 09:55:56,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a350e14933aa45f4a20ae9569593e7cc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/a350e14933aa45f4a20ae9569593e7cc 2023-04-07 09:55:56,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/42434b90131d4c528dde25620d01c2df to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/42434b90131d4c528dde25620d01c2df 2023-04-07 09:55:56,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81bfff3eb42d4a87bbe3c947947c619d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8764bd7e6cea48a4adf6c0520921adaf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2f3662b2d1394bbf8f3ae7fcb6896ec3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8f0d6af560f442193902223365dcbde, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/18b10f8de2374b10a9e3dd3b12356cbd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b2823b7cebf246a9a68c32616537cf41, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ee3eb0ebe0f040ada4aede7126a96c3c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/4133f05a28bb4e37b9686c6806468538, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8811fde885043e2a3eb2cf98bf8137d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2bc6e330408044c9ade5c689d4a69a6b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2871621eac674036850ffd4bfe5d8bad, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/46075c6648e24fbb82538b3db992f27c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7e4626c1a3cf4ff39773d875269a0600, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d45acbd8456b40cfa09dbad872cb4b72, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/597fffac677848ab97bb89d9ef8e7db1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b3139898f7564b949c3ea5953e26b59e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/6ab8e51d5c5044dcbd42d5b9726959e9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7aec0b8e65bf46919eec66e758e595f8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ca66f2f074514410bb1a717e79328b23, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8471f160da924ac99d64d2b4f2938370, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/156a39f76fa944ea84de49c9e4599b30, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/48d9c26201c04305ad5a4cab2a5aa89e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/93541b2a7b8e4f09993b63ef6420c244, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2424a88914384be7a28165294ad83a6d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81f667710fdb47b29c57d7f751ce9450, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/60673f91a3164d4f94f1148407c5d48d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/989fe31637d44eb5880c647fdf016928, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/870445cb74754f6db245034f5a09c6ce, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/3feb6c7c530f4b03820ddcbdf265c02c] to archive 2023-04-07 09:55:56,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:55:56,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81bfff3eb42d4a87bbe3c947947c619d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81bfff3eb42d4a87bbe3c947947c619d 2023-04-07 09:55:56,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8764bd7e6cea48a4adf6c0520921adaf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8764bd7e6cea48a4adf6c0520921adaf 2023-04-07 09:55:56,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2f3662b2d1394bbf8f3ae7fcb6896ec3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2f3662b2d1394bbf8f3ae7fcb6896ec3 2023-04-07 09:55:56,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8f0d6af560f442193902223365dcbde to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8f0d6af560f442193902223365dcbde 2023-04-07 09:55:56,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/18b10f8de2374b10a9e3dd3b12356cbd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/18b10f8de2374b10a9e3dd3b12356cbd 2023-04-07 09:55:56,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b2823b7cebf246a9a68c32616537cf41 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b2823b7cebf246a9a68c32616537cf41 2023-04-07 09:55:56,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/76d0c6d3c3354d6bb7a6a2a4aec7f3aa 2023-04-07 09:55:56,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ee3eb0ebe0f040ada4aede7126a96c3c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ee3eb0ebe0f040ada4aede7126a96c3c 2023-04-07 09:55:56,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/4133f05a28bb4e37b9686c6806468538 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/4133f05a28bb4e37b9686c6806468538 2023-04-07 09:55:56,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8811fde885043e2a3eb2cf98bf8137d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e8811fde885043e2a3eb2cf98bf8137d 2023-04-07 09:55:56,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2bc6e330408044c9ade5c689d4a69a6b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2bc6e330408044c9ade5c689d4a69a6b 2023-04-07 09:55:56,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2871621eac674036850ffd4bfe5d8bad to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2871621eac674036850ffd4bfe5d8bad 2023-04-07 09:55:56,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/46075c6648e24fbb82538b3db992f27c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/46075c6648e24fbb82538b3db992f27c 2023-04-07 09:55:56,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7e4626c1a3cf4ff39773d875269a0600 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7e4626c1a3cf4ff39773d875269a0600 2023-04-07 09:55:56,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d45acbd8456b40cfa09dbad872cb4b72 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/d45acbd8456b40cfa09dbad872cb4b72 2023-04-07 09:55:56,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/597fffac677848ab97bb89d9ef8e7db1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/597fffac677848ab97bb89d9ef8e7db1 2023-04-07 09:55:56,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b3139898f7564b949c3ea5953e26b59e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/b3139898f7564b949c3ea5953e26b59e 2023-04-07 09:55:56,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/6ab8e51d5c5044dcbd42d5b9726959e9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/6ab8e51d5c5044dcbd42d5b9726959e9 2023-04-07 09:55:56,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/1c3b9827a4fd4d6c8e2ae6a2cae72e04 2023-04-07 09:55:56,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7aec0b8e65bf46919eec66e758e595f8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/7aec0b8e65bf46919eec66e758e595f8 2023-04-07 09:55:56,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ca66f2f074514410bb1a717e79328b23 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/ca66f2f074514410bb1a717e79328b23 2023-04-07 09:55:56,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8471f160da924ac99d64d2b4f2938370 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8471f160da924ac99d64d2b4f2938370 2023-04-07 09:55:56,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/156a39f76fa944ea84de49c9e4599b30 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/156a39f76fa944ea84de49c9e4599b30 2023-04-07 09:55:56,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/48d9c26201c04305ad5a4cab2a5aa89e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/48d9c26201c04305ad5a4cab2a5aa89e 2023-04-07 09:55:56,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/93541b2a7b8e4f09993b63ef6420c244 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/93541b2a7b8e4f09993b63ef6420c244 2023-04-07 09:55:56,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2424a88914384be7a28165294ad83a6d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/2424a88914384be7a28165294ad83a6d 2023-04-07 09:55:56,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81f667710fdb47b29c57d7f751ce9450 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/81f667710fdb47b29c57d7f751ce9450 2023-04-07 09:55:56,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/60673f91a3164d4f94f1148407c5d48d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/60673f91a3164d4f94f1148407c5d48d 2023-04-07 09:55:56,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/989fe31637d44eb5880c647fdf016928 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/989fe31637d44eb5880c647fdf016928 2023-04-07 09:55:56,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/870445cb74754f6db245034f5a09c6ce to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/870445cb74754f6db245034f5a09c6ce 2023-04-07 09:55:56,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/3feb6c7c530f4b03820ddcbdf265c02c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/3feb6c7c530f4b03820ddcbdf265c02c 2023-04-07 09:55:56,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/recovered.edits/648.seqid, newMaxSeqId=648, maxSeqId=1 2023-04-07 09:55:56,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2. 2023-04-07 09:55:56,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c05568a99c8bbb1b0521e425f41a54f2: 2023-04-07 09:55:56,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:56,673 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=c05568a99c8bbb1b0521e425f41a54f2, regionState=CLOSED 2023-04-07 09:55:56,673 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861356673"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861356673"}]},"ts":"1680861356673"} 2023-04-07 09:55:56,675 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-07 09:55:56,675 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure c05568a99c8bbb1b0521e425f41a54f2, server=jenkins-hbase4.apache.org,35881,1680861218288 in 2.1520 sec 2023-04-07 09:55:56,676 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-07 09:55:56,676 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c05568a99c8bbb1b0521e425f41a54f2, UNASSIGN in 2.1570 sec 2023-04-07 09:55:56,677 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861356677"}]},"ts":"1680861356677"} 2023-04-07 09:55:56,677 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 09:55:56,679 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 09:55:56,680 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1690 sec 2023-04-07 09:55:58,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-07 09:55:58,620 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-07 09:55:58,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 09:55:58,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:58,622 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:58,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-07 09:55:58,623 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:58,627 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:58,628 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/recovered.edits] 2023-04-07 09:55:58,630 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c7a365e0a75743ff841148f52cc3fbfd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/c7a365e0a75743ff841148f52cc3fbfd 2023-04-07 09:55:58,631 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ca242b6be6b043389dc393ceada94634 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/ca242b6be6b043389dc393ceada94634 2023-04-07 09:55:58,632 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d519c7017e4f4e82b6234d5a5fe8b34f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/A/d519c7017e4f4e82b6234d5a5fe8b34f 2023-04-07 09:55:58,634 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/6a23604a299b475d9638df3e86a36dda to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/6a23604a299b475d9638df3e86a36dda 2023-04-07 09:55:58,635 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8b14f4fb3b3c400b97e7093a5e2a7ebe to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/8b14f4fb3b3c400b97e7093a5e2a7ebe 2023-04-07 09:55:58,635 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/ddcc667e676646678f1bbedfa33dd2fc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/B/ddcc667e676646678f1bbedfa33dd2fc 2023-04-07 09:55:58,637 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8ef51983a15d4e8694d0a613ea748585 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/8ef51983a15d4e8694d0a613ea748585 2023-04-07 09:55:58,638 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/a60b4f381d3a4e1ba911ab684f127c53 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/a60b4f381d3a4e1ba911ab684f127c53 2023-04-07 09:55:58,639 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e2c83df02d954a16b6b8f27a4b52302f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/C/e2c83df02d954a16b6b8f27a4b52302f 2023-04-07 09:55:58,640 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/recovered.edits/648.seqid to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2/recovered.edits/648.seqid 2023-04-07 09:55:58,641 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/c05568a99c8bbb1b0521e425f41a54f2 2023-04-07 09:55:58,641 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:55:58,642 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:58,643 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 09:55:58,645 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 09:55:58,646 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:58,646 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 09:55:58,646 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680861358646"}]},"ts":"9223372036854775807"} 2023-04-07 09:55:58,647 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 09:55:58,647 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c05568a99c8bbb1b0521e425f41a54f2, NAME => 'TestAcidGuarantees,,1680861332026.c05568a99c8bbb1b0521e425f41a54f2.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 09:55:58,647 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 09:55:58,647 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680861358647"}]},"ts":"9223372036854775807"} 2023-04-07 09:55:58,648 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 09:55:58,650 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:58,651 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 30 msec 2023-04-07 09:55:58,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-07 09:55:58,724 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-07 09:55:58,735 INFO [Listener at localhost/39511] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 228), OpenFileDescriptor=587 (was 554) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=404 (was 453), ProcessCount=169 (was 169), AvailableMemoryMB=2518 (was 2563) 2023-04-07 09:55:58,747 INFO [Listener at localhost/39511] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=587, MaxFileDescriptor=60000, SystemLoadAverage=404, ProcessCount=169, AvailableMemoryMB=2518 2023-04-07 09:55:58,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:55:58,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:55:58,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-07 09:55:58,751 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-07 09:55:58,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-07 09:55:58,751 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-07 09:55:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 09:55:58,752 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:58,753 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df empty. 2023-04-07 09:55:58,753 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:58,753 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:55:58,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 09:55:59,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 09:55:59,160 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:55:59,161 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 448acd9b275d13bcb18a1b50f0aa55df, NAME => 'TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp 2023-04-07 09:55:59,168 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:55:59,168 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 448acd9b275d13bcb18a1b50f0aa55df, disabling compactions & flushes 2023-04-07 09:55:59,168 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,168 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,168 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. after waiting 0 ms 2023-04-07 09:55:59,168 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,168 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,168 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:55:59,170 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-07 09:55:59,171 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861359170"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861359170"}]},"ts":"1680861359170"} 2023-04-07 09:55:59,171 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-07 09:55:59,172 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-07 09:55:59,172 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861359172"}]},"ts":"1680861359172"} 2023-04-07 09:55:59,172 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-07 09:55:59,176 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, ASSIGN}] 2023-04-07 09:55:59,177 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, ASSIGN 2023-04-07 09:55:59,178 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=false 2023-04-07 09:55:59,329 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:59,329 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861359329"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861359329"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861359329"}]},"ts":"1680861359329"} 2023-04-07 09:55:59,330 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:55:59,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 09:55:59,484 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 448acd9b275d13bcb18a1b50f0aa55df, NAME => 'TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:55:59,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:55:59,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,486 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,487 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A 2023-04-07 09:55:59,487 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A 2023-04-07 09:55:59,487 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:59,487 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 448acd9b275d13bcb18a1b50f0aa55df columnFamilyName A 2023-04-07 09:55:59,488 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(310): Store=448acd9b275d13bcb18a1b50f0aa55df/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:59,488 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,489 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B 2023-04-07 09:55:59,489 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B 2023-04-07 09:55:59,489 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:59,489 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 448acd9b275d13bcb18a1b50f0aa55df columnFamilyName B 2023-04-07 09:55:59,489 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(310): Store=448acd9b275d13bcb18a1b50f0aa55df/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:59,489 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,490 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C 2023-04-07 09:55:59,490 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C 2023-04-07 09:55:59,490 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:55:59,490 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 448acd9b275d13bcb18a1b50f0aa55df columnFamilyName C 2023-04-07 09:55:59,491 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(310): Store=448acd9b275d13bcb18a1b50f0aa55df/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:55:59,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:55:59,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:55:59,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-07 09:55:59,496 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 448acd9b275d13bcb18a1b50f0aa55df; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61643878, jitterRate=-0.08143463730812073}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:55:59,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:55:59,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., pid=56, masterSystemTime=1680861359481 2023-04-07 09:55:59,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:55:59,498 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:55:59,498 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861359498"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861359498"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861359498"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861359498"}]},"ts":"1680861359498"} 2023-04-07 09:55:59,500 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-07 09:55:59,500 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 in 169 msec 2023-04-07 09:55:59,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-07 09:55:59,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, ASSIGN in 324 msec 2023-04-07 09:55:59,501 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-07 09:55:59,501 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861359501"}]},"ts":"1680861359501"} 2023-04-07 09:55:59,502 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-07 09:55:59,504 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-07 09:55:59,505 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 755 msec 2023-04-07 09:55:59,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-07 09:55:59,855 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-07 09:55:59,856 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:55:59,861 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:55:59,862 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:55:59,863 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:55:59,864 DEBUG [Listener at localhost/39511] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-07 09:55:59,865 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-07 09:55:59,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-07 09:55:59,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-07 09:55:59,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:00,281 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 09:56:00,282 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-07 09:56:00,282 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-07 09:56:00,283 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-07 09:56:00,285 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, REOPEN/MOVE}] 2023-04-07 09:56:00,286 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, REOPEN/MOVE 2023-04-07 09:56:00,286 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,287 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861360286"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861360286"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861360286"}]},"ts":"1680861360286"} 2023-04-07 09:56:00,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:56:00,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 448acd9b275d13bcb18a1b50f0aa55df, disabling compactions & flushes 2023-04-07 09:56:00,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. after waiting 0 ms 2023-04-07 09:56:00,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-07 09:56:00,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:00,443 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 448acd9b275d13bcb18a1b50f0aa55df to self. 2023-04-07 09:56:00,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,445 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=CLOSED 2023-04-07 09:56:00,445 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861360445"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861360445"}]},"ts":"1680861360445"} 2023-04-07 09:56:00,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-07 09:56:00,447 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 in 159 msec 2023-04-07 09:56:00,447 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35881,1680861218288; forceNewPlan=false, retain=true 2023-04-07 09:56:00,598 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,598 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861360598"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861360598"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861360598"}]},"ts":"1680861360598"} 2023-04-07 09:56:00,599 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:56:00,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 448acd9b275d13bcb18a1b50f0aa55df, NAME => 'TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.', STARTKEY => '', ENDKEY => ''} 2023-04-07 09:56:00,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-07 09:56:00,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,754 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,755 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A 2023-04-07 09:56:00,755 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A 2023-04-07 09:56:00,755 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:56:00,755 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 448acd9b275d13bcb18a1b50f0aa55df columnFamilyName A 2023-04-07 09:56:00,756 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(310): Store=448acd9b275d13bcb18a1b50f0aa55df/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:56:00,757 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,757 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B 2023-04-07 09:56:00,757 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B 2023-04-07 09:56:00,757 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:56:00,758 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 448acd9b275d13bcb18a1b50f0aa55df columnFamilyName B 2023-04-07 09:56:00,758 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(310): Store=448acd9b275d13bcb18a1b50f0aa55df/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:56:00,758 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,758 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C 2023-04-07 09:56:00,758 DEBUG [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C 2023-04-07 09:56:00,759 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-07 09:56:00,759 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 448acd9b275d13bcb18a1b50f0aa55df columnFamilyName C 2023-04-07 09:56:00,759 INFO [StoreOpener-448acd9b275d13bcb18a1b50f0aa55df-1] regionserver.HStore(310): Store=448acd9b275d13bcb18a1b50f0aa55df/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-07 09:56:00,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-07 09:56:00,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 448acd9b275d13bcb18a1b50f0aa55df; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70918716, jitterRate=0.05677121877670288}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-07 09:56:00,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:00,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., pid=61, masterSystemTime=1680861360750 2023-04-07 09:56:00,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,765 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,765 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861360765"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680861360765"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680861360765"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680861360765"}]},"ts":"1680861360765"} 2023-04-07 09:56:00,767 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-07 09:56:00,767 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 in 167 msec 2023-04-07 09:56:00,768 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-07 09:56:00,768 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, REOPEN/MOVE in 482 msec 2023-04-07 09:56:00,769 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-07 09:56:00,769 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 485 msec 2023-04-07 09:56:00,770 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 903 msec 2023-04-07 09:56:00,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-07 09:56:00,772 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,776 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,777 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,780 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,780 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,784 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,784 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,788 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,788 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,792 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,792 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,801 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,801 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,805 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,805 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,813 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,814 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,817 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,818 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(139): Connect 0x0eedd898 to 127.0.0.1:49660 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-07 09:56:00,822 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@308b3f39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-07 09:56:00,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:00,828 DEBUG [hconnection-0x5e57c89c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,837 DEBUG [hconnection-0x3ec92327-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,837 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38308, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,837 DEBUG [hconnection-0x1683ad51-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,837 DEBUG [hconnection-0x455451f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,837 DEBUG [hconnection-0x67052c0a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,838 DEBUG [hconnection-0x6b08646f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,838 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,839 DEBUG [hconnection-0x5aa97c63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,839 DEBUG [hconnection-0x16b399bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:00,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:00,839 DEBUG [hconnection-0x740ae7a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,840 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38352, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,841 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38334, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,840 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:00,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:00,841 DEBUG [hconnection-0x4a3a5fb8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-07 09:56:00,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:00,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:00,842 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38382, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,842 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:00,844 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:00,844 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,845 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38396, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,845 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,846 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-07 09:56:00,846 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:00,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:00,846 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:00,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,847 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:00,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:00,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:00,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:00,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:00,850 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:00,850 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:00,850 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:00,850 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:00,852 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-07 09:56:00,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:00,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:00,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:00,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:00,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:00,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:00,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:00,860 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:00,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861360868, max timestamp=1680861360879Num uniques 859; 2023-04-07 09:56:00,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:56:00,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680861360868, max timestamp=1680861360879Num uniques 859; 2023-04-07 09:56:00,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861360868, max timestamp=1680861360879Num uniques 859; 2023-04-07 09:56:00,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:00,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-07 09:56:00,905 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a9f691a664924e8c91948d518889f79c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a9f691a664924e8c91948d518889f79c 2023-04-07 09:56:00,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680861360897, max timestamp=1680861360905Num uniques 1018; 2023-04-07 09:56:00,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680861360897, max timestamp=1680861360905Num uniques 1009; 2023-04-07 09:56:00,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680861360897, max timestamp=1680861360905Num uniques 1009; 2023-04-07 09:56:00,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-07 09:56:00,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-07 09:56:00,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-07 09:56:00,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:00,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861420923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861420924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861420926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:00,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861420926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:00,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:00,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861420927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861421028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861421030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861421030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861421030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861421032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861421232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861421233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861421233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861421234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861421235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,333 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/6479e0e3a862439b84eecfad30e501dd 2023-04-07 09:56:01,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861421537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861421537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861421538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861421539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:01,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861421541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:01,746 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/affdfdba46374a24b22840352c5a02e6 2023-04-07 09:56:02,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861422044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861422044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861422044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:02,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861422046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:02,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861422048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,162 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc 2023-04-07 09:56:02,166 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/6479e0e3a862439b84eecfad30e501dd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6479e0e3a862439b84eecfad30e501dd 2023-04-07 09:56:02,170 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6479e0e3a862439b84eecfad30e501dd, entries=150, sequenceid=12, filesize=25.1 K 2023-04-07 09:56:02,170 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/affdfdba46374a24b22840352c5a02e6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/affdfdba46374a24b22840352c5a02e6 2023-04-07 09:56:02,173 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/affdfdba46374a24b22840352c5a02e6, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 09:56:02,174 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc 2023-04-07 09:56:02,177 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc, entries=150, sequenceid=12, filesize=11.6 K 2023-04-07 09:56:02,178 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 448acd9b275d13bcb18a1b50f0aa55df in 1326ms, sequenceid=12, compaction requested=false 2023-04-07 09:56:02,178 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-07 09:56:02,178 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:02,178 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:02,179 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24039368[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:02,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:02,182 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:02,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,184 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:02,186 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:02,188 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:02,188 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:02,188 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:02,188 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:02,188 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:02,190 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,190 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,190 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:02,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,192 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-07 09:56:02,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:02,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:02,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:02,218 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,218 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:02,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:02,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:02,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:02,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:02,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:02,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:02,220 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:02,220 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:02,321 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:02,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:02,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:02,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:02,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:02,324 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:02,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:02,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:02,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:02,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,325 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:02,326 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:02,326 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:02,327 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:02,327 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:02,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:02,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:02,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:02,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:02,328 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:02,329 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-07 09:56:02,329 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:02,329 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:02,329 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:02,329 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:02,329 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:02,329 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:02,370 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407f6b065379bef45f3adf0b339ec9f66ee to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f6b065379bef45f3adf0b339ec9f66ee 2023-04-07 09:56:02,842 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=66, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/6b771ef0930f4cafbc86d7373cde8838 2023-04-07 09:56:02,878 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/860a6f060c524799b51f1c1aa80967bd 2023-04-07 09:56:03,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:03,053 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:03,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861423056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861423057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861423058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861423058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861423058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861423160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861423161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861423161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861423161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,295 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/9777a0b2a40842af99939ef1549808c8 2023-04-07 09:56:03,301 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/6b771ef0930f4cafbc86d7373cde8838 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6b771ef0930f4cafbc86d7373cde8838 2023-04-07 09:56:03,305 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6b771ef0930f4cafbc86d7373cde8838, entries=150, sequenceid=66, filesize=25.1 K 2023-04-07 09:56:03,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/860a6f060c524799b51f1c1aa80967bd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/860a6f060c524799b51f1c1aa80967bd 2023-04-07 09:56:03,313 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/860a6f060c524799b51f1c1aa80967bd, entries=150, sequenceid=66, filesize=11.6 K 2023-04-07 09:56:03,314 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/9777a0b2a40842af99939ef1549808c8 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/9777a0b2a40842af99939ef1549808c8 2023-04-07 09:56:03,318 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/9777a0b2a40842af99939ef1549808c8, entries=150, sequenceid=66, filesize=11.6 K 2023-04-07 09:56:03,318 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 448acd9b275d13bcb18a1b50f0aa55df in 989ms, sequenceid=66, compaction requested=false 2023-04-07 09:56:03,318 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:03,319 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:03,319 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:03,320 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,321 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,321 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:03,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f9ded03[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:03,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:03,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,323 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,324 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:03,324 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:03,324 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:03,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,324 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:03,325 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:03,325 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,325 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,325 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:03,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:03,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:03,328 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:03,328 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:03,328 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:03,328 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:03,328 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:03,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,339 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,340 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,340 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:03,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:03,340 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:03,340 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:03,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:03,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:03,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:56:03,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:03,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:03,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:03,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:03,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:03,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:03,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861363378, max timestamp=1680861363384Num uniques 859; 2023-04-07 09:56:03,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:56:03,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861423383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680861363378, max timestamp=1680861363384Num uniques 859; 2023-04-07 09:56:03,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-07 09:56:03,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861363379, max timestamp=1680861363384Num uniques 859; 2023-04-07 09:56:03,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:56:03,401 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a3fcb05e37cd43738b1bc13cd7c03b85 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a3fcb05e37cd43738b1bc13cd7c03b85 2023-04-07 09:56:03,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861423409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861423410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861423413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,417 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=77, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/3fe134e780cf4f66945541f4807ca5a1 2023-04-07 09:56:03,442 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:03,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:03,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:03,476 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:03,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:03,476 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:03,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/ffe6d7ed702e4903893904c1bd96cfd7 2023-04-07 09:56:03,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,478 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:03,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:03,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861423487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,489 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:03,492 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:03,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:03,493 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:03,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,503 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,503 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496273c2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:03,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:03,507 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,509 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:03,509 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:03,509 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:03,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,511 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:03,512 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861423514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:03,515 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:03,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:03,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:03,515 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:03,515 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:03,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,517 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,518 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861423516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861423517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,519 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:03,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:03,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:03,522 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,523 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,523 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:03,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,523 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,523 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:03,523 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,523 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,523 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,623 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:03,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:03,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:03,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:03,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:03,633 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:03,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:03,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:03,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,633 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,635 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,635 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,636 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,636 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:03,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:03,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:03,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:03,640 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:03,640 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,640 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:03,643 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:03,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,646 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:03,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bb13dcb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:03,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:03,648 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,650 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:03,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:03,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:03,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:03,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,653 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:03,653 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:03,653 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:03,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,656 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,657 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,658 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:03,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:03,661 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,661 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,662 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:03,662 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:03,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861423689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861423716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861423720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:03,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861423720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,762 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:03,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:03,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:03,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:03,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:03,772 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:03,772 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,772 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:03,772 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:03,772 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,773 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,774 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,774 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,775 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:03,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:03,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:03,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:03,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:03,776 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:03,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:03,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:03,777 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:03,777 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:03,779 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,779 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,780 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:03,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:03,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43116906[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:03,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:03,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,782 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,782 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,782 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:03,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:03,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:03,782 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:03,784 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,784 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:03,784 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,784 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:03,786 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:03,786 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:03,786 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:03,786 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:03,786 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:03,789 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,789 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,789 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:03,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:03,796 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,796 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,797 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:03,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,797 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:03,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,897 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:03,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:03,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:03,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:03,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:03,905 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:03,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:03,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:03,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,907 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,907 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/d0b8f76e28df43bb92ef13e80510e7d1 2023-04-07 09:56:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,908 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,908 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:03,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:03,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:03,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:03,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:03,909 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:03,910 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,910 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:03,910 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:03,910 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:03,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:03,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:03,910 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:03,911 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,912 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/3fe134e780cf4f66945541f4807ca5a1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/3fe134e780cf4f66945541f4807ca5a1 2023-04-07 09:56:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23847192[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:03,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:03,914 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,915 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:03,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:03,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/3fe134e780cf4f66945541f4807ca5a1, entries=150, sequenceid=77, filesize=25.1 K 2023-04-07 09:56:03,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/ffe6d7ed702e4903893904c1bd96cfd7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffe6d7ed702e4903893904c1bd96cfd7 2023-04-07 09:56:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:03,919 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffe6d7ed702e4903893904c1bd96cfd7, entries=150, sequenceid=77, filesize=11.6 K 2023-04-07 09:56:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/d0b8f76e28df43bb92ef13e80510e7d1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/d0b8f76e28df43bb92ef13e80510e7d1 2023-04-07 09:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:03,923 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:03,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:03,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:03,923 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:03,923 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:03,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,925 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:03,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:03,925 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,927 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/d0b8f76e28df43bb92ef13e80510e7d1, entries=150, sequenceid=77, filesize=11.6 K 2023-04-07 09:56:03,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 448acd9b275d13bcb18a1b50f0aa55df in 564ms, sequenceid=77, compaction requested=true 2023-04-07 09:56:03,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:03,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:03,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:03,929 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:56:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:03,929 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:03,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:03,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,930 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:03,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:03,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:03,929 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:56:03,930 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:03,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:03,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:03,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:03,931 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:03,931 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:03,931 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:03,931 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:56:03,931 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/A is initiating minor compaction (all files) 2023-04-07 09:56:03,931 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/A in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,931 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6479e0e3a862439b84eecfad30e501dd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6b771ef0930f4cafbc86d7373cde8838, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/3fe134e780cf4f66945541f4807ca5a1] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=75.4 K 2023-04-07 09:56:03,932 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 6479e0e3a862439b84eecfad30e501dd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680861360847 2023-04-07 09:56:03,932 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:56:03,932 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/B is initiating minor compaction (all files) 2023-04-07 09:56:03,932 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/B in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:03,932 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/affdfdba46374a24b22840352c5a02e6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/860a6f060c524799b51f1c1aa80967bd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffe6d7ed702e4903893904c1bd96cfd7] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=34.8 K 2023-04-07 09:56:03,932 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 6b771ef0930f4cafbc86d7373cde8838, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680861360922 2023-04-07 09:56:03,932 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting affdfdba46374a24b22840352c5a02e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680861360847 2023-04-07 09:56:03,933 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 3fe134e780cf4f66945541f4807ca5a1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861363363 2023-04-07 09:56:03,933 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 860a6f060c524799b51f1c1aa80967bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680861360922 2023-04-07 09:56:03,934 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ffe6d7ed702e4903893904c1bd96cfd7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861363363 2023-04-07 09:56:03,941 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#B#compaction#597 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:03,945 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#A#compaction#596 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:03,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:03,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=136.42 KB heapSize=380.69 KB 2023-04-07 09:56:03,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:03,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:03,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:03,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:03,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:03,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:04,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861424019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861424021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861424023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861424023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,031 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:04,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:04,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:04,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:04,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:04,035 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:04,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:04,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:04,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,035 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:04,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:04,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:04,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:04,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:04,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:04,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:04,038 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:04,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:04,039 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:04,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:04,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7922fa43[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:04,042 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:04,042 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,044 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:04,044 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,044 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:04,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:04,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:04,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:04,049 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:04,049 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:04,051 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,051 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,051 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,052 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,053 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,056 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:04,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:04,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,056 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:04,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,056 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:04,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861424125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,157 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:04,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:04,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:04,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:04,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:04,160 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:04,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:04,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:04,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,162 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:04,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:04,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:04,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:04,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:04,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:04,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:04,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0fdfc9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:04,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:04,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:04,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:04,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:04,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:04,180 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:04,185 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:04,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:04,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:04,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:04,185 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:04,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,187 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,188 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,189 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,192 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:04,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,192 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,192 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:04,192 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:04,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,292 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:04,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:04,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:04,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:04,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:04,296 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:04,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:04,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:04,296 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,297 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,301 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,301 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,301 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,301 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:04,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:04,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:04,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:04,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:04,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,305 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,305 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,305 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@431ecef7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:04,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:04,308 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:04,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:04,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:04,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:04,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:04,314 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:04,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:04,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:04,314 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:04,314 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:04,316 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,316 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,316 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,318 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,321 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,328 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:04,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:04,328 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:04,328 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:04,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:04,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861424331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,367 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/83a0df2f5f34480687a78852d42a8efd as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/83a0df2f5f34480687a78852d42a8efd 2023-04-07 09:56:04,372 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/B of 448acd9b275d13bcb18a1b50f0aa55df into 83a0df2f5f34480687a78852d42a8efd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:04,372 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:04,372 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/B, priority=13, startTime=1680861363929; duration=0sec 2023-04-07 09:56:04,372 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:04,372 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:56:04,373 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:56:04,373 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/C is initiating minor compaction (all files) 2023-04-07 09:56:04,373 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/C in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,373 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/9777a0b2a40842af99939ef1549808c8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/d0b8f76e28df43bb92ef13e80510e7d1] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=34.8 K 2023-04-07 09:56:04,374 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4f1fdaf5a5ae441fa6cab6db8c0a3dbc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680861360847 2023-04-07 09:56:04,374 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9777a0b2a40842af99939ef1549808c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680861360922 2023-04-07 09:56:04,374 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d0b8f76e28df43bb92ef13e80510e7d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861363363 2023-04-07 09:56:04,392 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#C#compaction#599 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:04,396 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/83aed4a4f4644315870bffae9708da8b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/83aed4a4f4644315870bffae9708da8b 2023-04-07 09:56:04,400 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/A of 448acd9b275d13bcb18a1b50f0aa55df into 83aed4a4f4644315870bffae9708da8b(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:04,401 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:04,401 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/A, priority=13, startTime=1680861363929; duration=0sec 2023-04-07 09:56:04,401 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:04,411 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/8213919df9d6459bb79546180e34fd7f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8213919df9d6459bb79546180e34fd7f 2023-04-07 09:56:04,411 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407ecccb72dbbbe435e92dda61ac6fd1537 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ecccb72dbbbe435e92dda61ac6fd1537 2023-04-07 09:56:04,418 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/C of 448acd9b275d13bcb18a1b50f0aa55df into 8213919df9d6459bb79546180e34fd7f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:04,418 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:04,418 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/C, priority=13, startTime=1680861363929; duration=0sec 2023-04-07 09:56:04,419 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:04,428 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:04,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:04,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:04,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:04,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:04,452 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:04,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:04,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:04,453 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,453 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,453 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/29f6e2f04fd545ba95094f690915399c 2023-04-07 09:56:04,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,455 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:04,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:04,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:04,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:04,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,468 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:04,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:04,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:04,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:04,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:04,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:04,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:04,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:04,485 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:04,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43963fe7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:04,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:04,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:04,491 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,491 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:04,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:04,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,493 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:04,493 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:04,496 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:04,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:04,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:04,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:04,497 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:04,502 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,502 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,502 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,502 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,505 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,508 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:04,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:04,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,508 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,509 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:04,509 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:04,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861424524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861424526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861424529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,609 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:04,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:04,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:04,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:04,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:04,616 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:04,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:04,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:04,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,616 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:04,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:04,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:04,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:04,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:04,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,619 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:04,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,622 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,623 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d882534[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:04,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:04,624 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,627 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:04,627 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,627 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:04,629 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:04,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,629 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:04,634 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:04,634 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:04,634 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:04,634 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:04,634 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:04,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:04,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861424634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,637 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,639 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,643 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:04,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:04,643 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:04,643 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:04,643 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,643 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,644 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:04,743 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:04,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:04,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:04,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:04,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:04,746 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,749 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,750 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:04,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:04,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:04,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:04,756 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:04,764 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,764 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:04,764 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:04,764 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:04,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:04,769 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,769 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:04,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588c56db[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:04,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:04,771 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:04,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:04,772 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:04,773 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,773 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:04,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:04,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,775 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,775 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:04,778 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:04,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:04,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:04,779 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:04,779 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:04,780 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,780 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,780 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,783 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,786 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,787 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:04,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:04,787 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:04,787 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:04,887 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:04,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:04,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:04,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:04,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:04,892 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:04,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:04,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:04,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,893 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/0115b1ac44014bd98e3732c1c1ff8136 2023-04-07 09:56:04,894 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,894 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,895 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,895 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:04,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:04,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:04,897 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,897 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:04,897 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:04,897 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:04,898 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,898 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:04,898 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:04,900 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd0424b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:04,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:04,903 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,904 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:04,904 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,905 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:04,906 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:04,906 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,906 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,906 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:04,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:04,909 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:04,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:04,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:04,909 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:04,909 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:04,911 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,912 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,912 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,913 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,917 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,917 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,917 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:04,917 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:04,917 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:04,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:04,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:04,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:04,917 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:04,918 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:04,918 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:04,917 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:04,917 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:04,918 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:04,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:04,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/b83c9f6378834e8290b9709d9fc5c947 2023-04-07 09:56:04,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/29f6e2f04fd545ba95094f690915399c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/29f6e2f04fd545ba95094f690915399c 2023-04-07 09:56:04,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/29f6e2f04fd545ba95094f690915399c, entries=200, sequenceid=112, filesize=31.9 K 2023-04-07 09:56:04,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/0115b1ac44014bd98e3732c1c1ff8136 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0115b1ac44014bd98e3732c1c1ff8136 2023-04-07 09:56:04,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0115b1ac44014bd98e3732c1c1ff8136, entries=150, sequenceid=112, filesize=11.6 K 2023-04-07 09:56:04,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/b83c9f6378834e8290b9709d9fc5c947 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b83c9f6378834e8290b9709d9fc5c947 2023-04-07 09:56:04,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b83c9f6378834e8290b9709d9fc5c947, entries=150, sequenceid=112, filesize=11.6 K 2023-04-07 09:56:04,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~380.64 KB/389776, currentSize=46.96 KB/48090 for 448acd9b275d13bcb18a1b50f0aa55df in 948ms, sequenceid=112, compaction requested=false 2023-04-07 09:56:04,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:05,018 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:05,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:05,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:05,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:05,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:05,023 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:05,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:05,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:05,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:05,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:05,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:05,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:05,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:05,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:05,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,026 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:05,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:05,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:05,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:05,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:05,027 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:05,027 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:05,028 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:05,028 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:56:05,028 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:05,029 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:05,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:05,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:05,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861365127, max timestamp=1680861365142Num uniques 859; 2023-04-07 09:56:05,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:05,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861365127, max timestamp=1680861365142Num uniques 859; 2023-04-07 09:56:05,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:05,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861365127, max timestamp=1680861365142Num uniques 859; 2023-04-07 09:56:05,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:05,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861425222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861425225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861425326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861425328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,495 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304079be05eba5b0a468f8507f227097efaaf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079be05eba5b0a468f8507f227097efaaf 2023-04-07 09:56:05,522 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/9fad943c543147baad71938b36692684 2023-04-07 09:56:05,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861425529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861425529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861425536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861425537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861425537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861425833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:05,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861425842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:05,946 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/4ad0e307a18645ce9a7b621158a80fbf 2023-04-07 09:56:05,966 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/f4caabc5a807423bb9cf09da455e4505 2023-04-07 09:56:05,973 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/9fad943c543147baad71938b36692684 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/9fad943c543147baad71938b36692684 2023-04-07 09:56:05,985 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/9fad943c543147baad71938b36692684, entries=150, sequenceid=125, filesize=25.1 K 2023-04-07 09:56:05,985 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/4ad0e307a18645ce9a7b621158a80fbf as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4ad0e307a18645ce9a7b621158a80fbf 2023-04-07 09:56:06,004 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4ad0e307a18645ce9a7b621158a80fbf, entries=150, sequenceid=125, filesize=11.6 K 2023-04-07 09:56:06,005 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/f4caabc5a807423bb9cf09da455e4505 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f4caabc5a807423bb9cf09da455e4505 2023-04-07 09:56:06,016 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f4caabc5a807423bb9cf09da455e4505, entries=150, sequenceid=125, filesize=11.6 K 2023-04-07 09:56:06,017 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 448acd9b275d13bcb18a1b50f0aa55df in 989ms, sequenceid=125, compaction requested=true 2023-04-07 09:56:06,017 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:06,017 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:06,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:06,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:06,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:06,017 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:06,019 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4b2111[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:06,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:06,021 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,022 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,022 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:06,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:06,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:06,023 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:06,024 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:06,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,025 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:06,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:06,027 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:06,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:06,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:06,027 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:06,027 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:06,030 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,030 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,030 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,031 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,033 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:06,036 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:06,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:06,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:06,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,036 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,036 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:06,037 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:06,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:06,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,137 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:06,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:06,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:06,140 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:06,140 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:06,140 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:06,140 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:06,141 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,141 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:06,143 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:06,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,144 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:06,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:06,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:06,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:06,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:06,150 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:06,152 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 09:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:06,152 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:06,153 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:06,153 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:06,153 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:06,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:06,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861426352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861426355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861426456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861426459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,569 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407f60cedcf9b49473eaf3accde7e4d3110 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f60cedcf9b49473eaf3accde7e4d3110 2023-04-07 09:56:06,575 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=162, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/8fcfb19c71c7437e9ac7b22f1b6ec95e 2023-04-07 09:56:06,604 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/7b980e29ee9d46788077dd19547cbdb6 2023-04-07 09:56:06,634 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/b5ff43efec764631b8334a3dfe1416c7 2023-04-07 09:56:06,637 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/8fcfb19c71c7437e9ac7b22f1b6ec95e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8fcfb19c71c7437e9ac7b22f1b6ec95e 2023-04-07 09:56:06,640 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8fcfb19c71c7437e9ac7b22f1b6ec95e, entries=150, sequenceid=162, filesize=25.3 K 2023-04-07 09:56:06,641 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/7b980e29ee9d46788077dd19547cbdb6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/7b980e29ee9d46788077dd19547cbdb6 2023-04-07 09:56:06,644 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/7b980e29ee9d46788077dd19547cbdb6, entries=150, sequenceid=162, filesize=11.8 K 2023-04-07 09:56:06,645 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/b5ff43efec764631b8334a3dfe1416c7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b5ff43efec764631b8334a3dfe1416c7 2023-04-07 09:56:06,649 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b5ff43efec764631b8334a3dfe1416c7, entries=150, sequenceid=162, filesize=11.8 K 2023-04-07 09:56:06,650 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 448acd9b275d13bcb18a1b50f0aa55df in 498ms, sequenceid=162, compaction requested=true 2023-04-07 09:56:06,650 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:06,650 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:06,650 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:06,652 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73061649[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:06,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:06,654 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,655 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:06,655 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,656 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:06,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:06,662 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,662 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,663 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:06,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:56:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:06,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:06,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:06,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:06,670 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:06,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:06,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:06,670 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:06,670 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:06,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,672 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,672 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,673 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,676 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:06,676 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:06,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,677 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,677 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:06,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:06,678 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:06,678 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:06,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:06,688 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e0312f8c39124354ba8c4667112bd3f7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e0312f8c39124354ba8c4667112bd3f7 2023-04-07 09:56:06,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861426698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861426701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,778 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:06,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:06,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:06,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:06,781 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:06,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:06,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:06,781 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:06,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,784 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,789 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:06,789 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:06,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,790 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:06,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:06,791 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:06,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:06,791 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:06,792 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:06,792 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:06,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:06,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:06,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:06,792 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:06,793 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,793 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,794 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4906ac7c[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:06,795 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:06,796 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,797 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:06,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:06,797 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:06,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,797 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:06,798 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:06,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,798 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:06,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:06,803 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:06,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:06,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:06,803 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:06,803 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:06,806 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,806 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,806 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,807 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861426802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:06,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861426805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,810 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:06,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:06,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:06,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:06,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:06,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,811 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:06,811 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:06,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:06,811 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,812 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,911 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:06,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:06,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:06,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:06,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:06,915 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:06,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:06,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:06,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:06,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,915 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,916 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,916 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:06,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:06,917 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,917 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:06,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:06,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:06,918 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:06,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:06,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:06,919 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:06,919 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:06,919 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:06,919 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:06,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:06,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:06,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:06,919 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:06,920 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,920 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,921 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,921 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:06,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262b7d0d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:06,922 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:06,922 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,924 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:06,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:06,924 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,924 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:06,924 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:06,926 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:06,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,926 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:06,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:06,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:06,928 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:06,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:06,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:06,928 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:06,928 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:06,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,929 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,930 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:06,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:06,930 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:06,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,931 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,933 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:06,933 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:06,933 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:06,933 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:06,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:06,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:06,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:06,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:06,933 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:06,933 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:06,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:06,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:06,933 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,934 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:06,934 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:06,934 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,934 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:06,934 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861427008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861427010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,033 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,037 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,041 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,042 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,042 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,042 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,043 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,043 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ae8370[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,045 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,045 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,046 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,046 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,047 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,047 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,047 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,048 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,048 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:07,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,050 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,050 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,050 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,055 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,056 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,057 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,059 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:07,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,059 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,060 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,060 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,097 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/bd0abd8935e34543a7834ab5e05afa05 2023-04-07 09:56:07,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/5d72254dea4844f1bb50bc8859dfafe1 2023-04-07 09:56:07,160 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,163 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,163 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,164 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,164 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,166 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,166 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,166 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,166 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,166 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,168 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23596572[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,169 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,169 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,172 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,173 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:07,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,175 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,175 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,176 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,177 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,181 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,183 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:07,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,183 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,183 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,183 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,284 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,286 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,286 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,287 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,289 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,289 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,289 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,289 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,291 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,291 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,291 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,291 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,293 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,293 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f404829[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,294 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,294 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,295 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,297 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,297 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,297 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,303 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,303 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,305 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,307 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:07,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,310 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,311 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861427313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861427314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,411 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,415 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,418 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,419 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,419 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,419 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,419 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,419 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,420 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,421 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,421 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,421 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,421 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@227e95d9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,423 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,423 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,425 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,426 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,426 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,427 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:07,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,430 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,430 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,430 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,430 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,430 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,432 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,433 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,434 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,444 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,445 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:07,445 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/087ce9702ac94c7bb2ab9c58c4726503 2023-04-07 09:56:07,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/bd0abd8935e34543a7834ab5e05afa05 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/bd0abd8935e34543a7834ab5e05afa05 2023-04-07 09:56:07,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/bd0abd8935e34543a7834ab5e05afa05, entries=200, sequenceid=174, filesize=32.1 K 2023-04-07 09:56:07,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/5d72254dea4844f1bb50bc8859dfafe1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/5d72254dea4844f1bb50bc8859dfafe1 2023-04-07 09:56:07,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/5d72254dea4844f1bb50bc8859dfafe1, entries=150, sequenceid=174, filesize=11.8 K 2023-04-07 09:56:07,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/087ce9702ac94c7bb2ab9c58c4726503 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/087ce9702ac94c7bb2ab9c58c4726503 2023-04-07 09:56:07,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/087ce9702ac94c7bb2ab9c58c4726503, entries=150, sequenceid=174, filesize=11.8 K 2023-04-07 09:56:07,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 448acd9b275d13bcb18a1b50f0aa55df in 866ms, sequenceid=174, compaction requested=true 2023-04-07 09:56:07,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:07,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:07,529 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:07,529 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:07,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:07,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:07,530 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143063 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:07,531 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/A is initiating minor compaction (all files) 2023-04-07 09:56:07,531 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:07,531 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/A in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,531 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/B is initiating minor compaction (all files) 2023-04-07 09:56:07,531 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/83aed4a4f4644315870bffae9708da8b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/29f6e2f04fd545ba95094f690915399c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/9fad943c543147baad71938b36692684, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8fcfb19c71c7437e9ac7b22f1b6ec95e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/bd0abd8935e34543a7834ab5e05afa05] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=139.7 K 2023-04-07 09:56:07,531 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/B in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,531 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/83a0df2f5f34480687a78852d42a8efd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0115b1ac44014bd98e3732c1c1ff8136, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4ad0e307a18645ce9a7b621158a80fbf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/7b980e29ee9d46788077dd19547cbdb6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/5d72254dea4844f1bb50bc8859dfafe1] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=58.4 K 2023-04-07 09:56:07,531 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 83aed4a4f4644315870bffae9708da8b, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861363363 2023-04-07 09:56:07,531 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 83a0df2f5f34480687a78852d42a8efd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861363363 2023-04-07 09:56:07,532 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 29f6e2f04fd545ba95094f690915399c, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680861363407 2023-04-07 09:56:07,532 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0115b1ac44014bd98e3732c1c1ff8136, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680861363407 2023-04-07 09:56:07,532 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 9fad943c543147baad71938b36692684, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861364011 2023-04-07 09:56:07,532 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4ad0e307a18645ce9a7b621158a80fbf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861364011 2023-04-07 09:56:07,532 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8fcfb19c71c7437e9ac7b22f1b6ec95e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861365208 2023-04-07 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 7b980e29ee9d46788077dd19547cbdb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861365208 2023-04-07 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting bd0abd8935e34543a7834ab5e05afa05, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680861366351 2023-04-07 09:56:07,533 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 5d72254dea4844f1bb50bc8859dfafe1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680861366354 2023-04-07 09:56:07,541 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#B#compaction#611 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:07,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861366691, max timestamp=1680861367541Num uniques 859; 2023-04-07 09:56:07,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861366691, max timestamp=1680861367541Num uniques 859; 2023-04-07 09:56:07,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:07,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861366691, max timestamp=1680861367541Num uniques 859; 2023-04-07 09:56:07,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:07,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:07,547 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#A#compaction#612 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:07,547 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,551 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,551 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:07,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-07 09:56:07,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:07,553 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:07,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:07,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:07,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:07,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,556 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,556 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,557 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,557 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,557 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,558 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/8a4b8e66de5a4540867c4cf933e652c3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8a4b8e66de5a4540867c4cf933e652c3 2023-04-07 09:56:07,558 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,558 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36df37e5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,563 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,563 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/B of 448acd9b275d13bcb18a1b50f0aa55df into 8a4b8e66de5a4540867c4cf933e652c3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:07,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,563 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:07,563 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/B, priority=11, startTime=1680861367529; duration=0sec 2023-04-07 09:56:07,563 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:07,564 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:07,565 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:07,565 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/C is initiating minor compaction (all files) 2023-04-07 09:56:07,565 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/C in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,565 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8213919df9d6459bb79546180e34fd7f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b83c9f6378834e8290b9709d9fc5c947, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f4caabc5a807423bb9cf09da455e4505, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b5ff43efec764631b8334a3dfe1416c7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/087ce9702ac94c7bb2ab9c58c4726503] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=58.4 K 2023-04-07 09:56:07,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,565 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,565 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8213919df9d6459bb79546180e34fd7f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1680861363363 2023-04-07 09:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,566 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b83c9f6378834e8290b9709d9fc5c947, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680861363407 2023-04-07 09:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,566 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting f4caabc5a807423bb9cf09da455e4505, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1680861364011 2023-04-07 09:56:07,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,566 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b5ff43efec764631b8334a3dfe1416c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680861365208 2023-04-07 09:56:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,567 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 087ce9702ac94c7bb2ab9c58c4726503, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680861366354 2023-04-07 09:56:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:07,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,568 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,568 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,568 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,573 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,575 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#C#compaction#614 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,577 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,577 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,578 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407a312ac9f5567400bb123c07f5a3bfb1e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a312ac9f5567400bb123c07f5a3bfb1e 2023-04-07 09:56:07,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861427596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861367590, max timestamp=1680861367600Num uniques 859; 2023-04-07 09:56:07,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861367590, max timestamp=1680861367600Num uniques 859; 2023-04-07 09:56:07,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861367590, max timestamp=1680861367600Num uniques 859; 2023-04-07 09:56:07,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:07,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:07,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:07,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861427660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861427661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,677 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,680 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,680 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,681 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,684 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,685 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,686 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,686 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,686 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,686 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,686 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,689 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e45a6b9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,693 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,694 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,695 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:07,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,699 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,699 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,699 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,701 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,702 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861427703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,708 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,709 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,709 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861427766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861427766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,809 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,813 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,813 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,814 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,815 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,816 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,818 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861427817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861427819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@319ac58[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,820 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,822 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,822 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,825 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:07,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,828 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,828 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,829 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,830 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,834 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,834 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,835 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861427906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,935 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:07,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:07,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:07,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:07,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:07,938 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:07,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:07,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:07,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,938 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,943 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,944 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:07,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:07,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:07,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:07,945 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:07,945 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:07,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:07,946 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:07,946 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:07,947 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,947 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:07,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559f21bf[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:07,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:07,949 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,950 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:07,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:07,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:07,951 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,951 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:07,952 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,952 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:07,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:07,954 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:07,954 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:07,954 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:07,954 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:07,955 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:07,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:07,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:07,960 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:07,964 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:07,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:07,964 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:07,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:07,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:07,964 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:07,964 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:07,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:07,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861427969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:07,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861427969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:07,996 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/40f03b4909bf47cda272b91b3ee31023 2023-04-07 09:56:08,001 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/4472b330a1ff48c3937db8210c0c6883 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4472b330a1ff48c3937db8210c0c6883 2023-04-07 09:56:08,005 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/C of 448acd9b275d13bcb18a1b50f0aa55df into 4472b330a1ff48c3937db8210c0c6883(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:08,005 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:08,006 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/C, priority=11, startTime=1680861367529; duration=0sec 2023-04-07 09:56:08,006 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:08,065 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:08,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:08,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:08,068 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:08,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:08,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:08,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,069 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,070 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,070 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,071 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:08,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:08,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:08,076 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:08,076 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,077 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:08,077 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:08,077 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:08,078 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,078 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab054f6[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:08,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:08,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,083 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,083 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:08,083 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:08,083 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:08,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,083 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:08,084 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,084 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:08,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:08,087 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:08,087 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:08,087 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:08,087 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:08,087 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:08,088 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,088 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,088 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,089 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,090 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:08,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:08,092 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,092 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:08,093 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:08,093 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:08,193 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:08,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:08,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:08,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:08,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:08,200 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:08,200 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,200 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:08,200 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:08,201 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,201 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,203 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,203 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,204 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,204 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:08,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:08,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:08,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,210 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:08,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:08,215 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:08,215 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,215 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:08,216 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:08,216 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:08,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861428211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:08,216 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:08,217 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:08,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b34977[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:08,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:08,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,221 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:08,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:08,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:08,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,221 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:08,222 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:08,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,223 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:08,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:08,225 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:08,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:08,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:08,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:08,225 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:08,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,228 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,231 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:08,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:08,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,233 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,233 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:08,233 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:08,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861428272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861428272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,333 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:08,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:08,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:08,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:08,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:08,337 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:08,337 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,337 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:08,337 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:08,337 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,337 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,338 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,339 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,339 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:08,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:08,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:08,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:08,340 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,340 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:08,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:08,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:08,341 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:08,346 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,346 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27065375[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:08,348 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:08,348 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,350 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:08,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:08,350 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:08,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,350 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:08,352 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:08,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:08,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:08,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:08,355 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:08,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:08,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:08,355 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:08,355 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:08,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,356 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,357 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:08,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:08,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,362 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,363 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,363 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,362 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:08,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:08,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,363 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:08,377 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/14a7cc3242174f04a71d58a7a6da3d9c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/14a7cc3242174f04a71d58a7a6da3d9c 2023-04-07 09:56:08,383 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/A of 448acd9b275d13bcb18a1b50f0aa55df into 14a7cc3242174f04a71d58a7a6da3d9c(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:08,383 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:08,383 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/A, priority=11, startTime=1680861367529; duration=0sec 2023-04-07 09:56:08,383 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:08,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/8e42dc7e180d4c529853a24f8a4fec83 2023-04-07 09:56:08,463 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:08,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:08,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:08,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:08,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:08,470 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:08,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:08,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:08,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,472 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,472 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:08,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:08,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:08,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:08,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:08,473 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:08,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:08,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:08,474 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:08,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:08,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b380cf3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:08,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:08,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,480 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:08,480 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,481 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:08,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:08,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:08,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:08,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:08,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:08,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:08,485 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:08,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,489 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:08,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,498 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:08,498 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,598 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:08,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:08,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:08,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:08,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:08,602 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:08,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:08,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:08,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,603 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,603 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:08,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:08,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:08,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:08,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:08,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:08,605 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:08,605 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,606 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:08,606 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:08,606 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:08,608 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13b7fe86[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:08,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:08,611 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:08,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:08,612 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:08,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:08,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:08,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:08,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:08,614 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:08,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:08,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:08,614 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:08,614 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:08,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,616 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,622 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:08,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,623 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,623 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,623 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:08,623 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:08,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861428717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,723 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:08,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:08,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:08,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:08,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:08,727 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,727 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,728 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,729 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:08,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:08,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:08,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:08,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:08,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:08,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:08,739 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:08,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:08,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:08,739 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:08,745 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,746 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,746 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60bce576[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:08,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:08,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,749 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:08,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:08,749 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:08,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,749 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:08,754 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:08,754 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,754 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:08,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:08,757 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:08,757 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:08,757 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:08,757 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:08,757 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:08,760 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,760 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,760 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:08,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:08,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,768 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:08,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:08,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,769 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:08,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:08,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:08,772 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:08,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:08,772 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:08,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:08,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:08,773 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:08,773 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:08,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:08,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861428779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861428781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/cb6c464667f44f33ab9a7e6404f1b07e 2023-04-07 09:56:08,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/40f03b4909bf47cda272b91b3ee31023 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/40f03b4909bf47cda272b91b3ee31023 2023-04-07 09:56:08,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/40f03b4909bf47cda272b91b3ee31023, entries=150, sequenceid=199, filesize=25.3 K 2023-04-07 09:56:08,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/8e42dc7e180d4c529853a24f8a4fec83 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8e42dc7e180d4c529853a24f8a4fec83 2023-04-07 09:56:08,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8e42dc7e180d4c529853a24f8a4fec83, entries=150, sequenceid=199, filesize=11.8 K 2023-04-07 09:56:08,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/cb6c464667f44f33ab9a7e6404f1b07e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/cb6c464667f44f33ab9a7e6404f1b07e 2023-04-07 09:56:08,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861428828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:08,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861428828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/cb6c464667f44f33ab9a7e6404f1b07e, entries=150, sequenceid=199, filesize=11.8 K 2023-04-07 09:56:08,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for 448acd9b275d13bcb18a1b50f0aa55df in 1284ms, sequenceid=199, compaction requested=false 2023-04-07 09:56:08,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:08,873 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:08,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:08,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:08,876 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:08,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:08,876 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:08,877 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,877 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,878 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:08,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:08,878 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:08,879 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:08,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:08,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:08,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:08,880 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:08,880 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:08,880 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 09:56:08,881 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:08,881 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:08,881 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:08,881 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:08,881 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:08,881 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:08,901 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304077187bd523427403d979a5d5de46e7f59 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077187bd523427403d979a5d5de46e7f59 2023-04-07 09:56:09,315 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/abe9e3a808344d27889febc59f673d81 2023-04-07 09:56:09,326 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/03eef0fec755435db9cb23053b44c6bb 2023-04-07 09:56:09,335 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/fb1dbb49de3744999342d29edf108542 2023-04-07 09:56:09,339 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/abe9e3a808344d27889febc59f673d81 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/abe9e3a808344d27889febc59f673d81 2023-04-07 09:56:09,342 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/abe9e3a808344d27889febc59f673d81, entries=150, sequenceid=239, filesize=25.3 K 2023-04-07 09:56:09,343 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/03eef0fec755435db9cb23053b44c6bb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/03eef0fec755435db9cb23053b44c6bb 2023-04-07 09:56:09,347 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/03eef0fec755435db9cb23053b44c6bb, entries=150, sequenceid=239, filesize=11.8 K 2023-04-07 09:56:09,347 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/fb1dbb49de3744999342d29edf108542 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/fb1dbb49de3744999342d29edf108542 2023-04-07 09:56:09,352 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/fb1dbb49de3744999342d29edf108542, entries=150, sequenceid=239, filesize=11.8 K 2023-04-07 09:56:09,353 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 448acd9b275d13bcb18a1b50f0aa55df in 472ms, sequenceid=239, compaction requested=true 2023-04-07 09:56:09,353 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:09,353 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:09,355 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,356 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,356 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:09,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b48fef[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:09,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:09,358 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,359 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:09,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:09,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:09,359 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:09,360 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,360 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:09,363 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:09,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:09,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:09,363 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:09,363 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:09,365 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,365 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,365 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,367 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:09,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:09,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,371 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,371 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,471 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:09,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:09,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:09,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:09,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:09,475 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:09,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:09,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:09,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,475 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,477 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:09,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:09,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:09,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:09,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,478 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:09,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:09,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,484 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:09,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e205a3e[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:09,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:09,487 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:09,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:09,491 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:09,495 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,496 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:09,498 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:09,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:09,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:09,498 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:09,498 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:09,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,501 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,502 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,504 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:09,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:09,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:09,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,505 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:09,505 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:09,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,605 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:09,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:09,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:09,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:09,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:09,610 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:09,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:09,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:09,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,610 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:09,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:09,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:09,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:09,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:09,614 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,615 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:09,615 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:09,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@124fd0b2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:09,619 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:09,619 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,620 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,621 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:09,621 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:09,621 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:09,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,621 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:09,626 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:09,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:09,629 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:09,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:09,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:09,629 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:09,629 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:09,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,636 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,636 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,638 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,641 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:09,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:09,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,641 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,641 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:09,641 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:09,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,646 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:09,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,756 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:09,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:09,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:09,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:09,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:09,761 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:09,761 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,761 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:09,761 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:09,761 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,761 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,762 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,762 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,763 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,763 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:09,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:09,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:09,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:09,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,764 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:09,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:09,768 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,768 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:09,768 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:09,769 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-07 09:56:09,769 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:09,769 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:09,769 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:09,769 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:09,769 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:09,769 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:09,781 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407864241dc0f2d48a0b15d724f9600cde3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407864241dc0f2d48a0b15d724f9600cde3 2023-04-07 09:56:09,798 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/5e3e58b934264c8d9634a1d65a87f918 2023-04-07 09:56:09,816 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/a957bb862fd4447d8f55562eb005a0c0 2023-04-07 09:56:09,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:09,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:09,834 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/68359d9e4a9f44e28c4b0c766619a3c6 2023-04-07 09:56:09,838 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/5e3e58b934264c8d9634a1d65a87f918 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5e3e58b934264c8d9634a1d65a87f918 2023-04-07 09:56:09,842 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5e3e58b934264c8d9634a1d65a87f918, entries=50, sequenceid=245, filesize=11.6 K 2023-04-07 09:56:09,842 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/a957bb862fd4447d8f55562eb005a0c0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/a957bb862fd4447d8f55562eb005a0c0 2023-04-07 09:56:09,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1680861369786, max timestamp=1680861369842Num uniques 859; 2023-04-07 09:56:09,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1680861369786, max timestamp=1680861369842Num uniques 859; 2023-04-07 09:56:09,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-07 09:56:09,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-07 09:56:09,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1680861369786, max timestamp=1680861369842Num uniques 859; 2023-04-07 09:56:09,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-04-07 09:56:09,846 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/a957bb862fd4447d8f55562eb005a0c0, entries=50, sequenceid=245, filesize=7.1 K 2023-04-07 09:56:09,848 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/68359d9e4a9f44e28c4b0c766619a3c6 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/68359d9e4a9f44e28c4b0c766619a3c6 2023-04-07 09:56:09,853 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/68359d9e4a9f44e28c4b0c766619a3c6, entries=50, sequenceid=245, filesize=7.1 K 2023-04-07 09:56:09,854 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=147.60 KB/151140 for 448acd9b275d13bcb18a1b50f0aa55df in 86ms, sequenceid=245, compaction requested=true 2023-04-07 09:56:09,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:09,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:09,857 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,859 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:09,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155f10d8[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:09,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:09,861 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:09,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=154.31 KB heapSize=388.50 KB 2023-04-07 09:56:09,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:09,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:09,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:09,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:09,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:09,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:09,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,862 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:09,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:09,863 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:09,864 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,864 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:09,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:09,867 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:09,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:09,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:09,867 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:09,867 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:09,869 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,869 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,869 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,869 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,874 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,874 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:09,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:09,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:09,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,874 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:09,874 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:09,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:09,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861429873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:09,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861429875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:09,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861429877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,924 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304070f11447944df4a83a02794c5d0e0ff86 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070f11447944df4a83a02794c5d0e0ff86 2023-04-07 09:56:09,975 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:09,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:09,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:09,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:09,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:09,979 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:09,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:09,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:09,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:09,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861429978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,980 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:09,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:09,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:09,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861429980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:09,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:09,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:09,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:09,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:09,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:09,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:09,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861429981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,985 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:09,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c53e6b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:09,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:09,987 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,988 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:09,988 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:09,989 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,989 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:09,991 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:09,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:09,994 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:09,994 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:09,994 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:09,994 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:09,994 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:09,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,996 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:09,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,997 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:09,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:09,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:09,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:10,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:10,001 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,001 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,002 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:10,002 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:10,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:10,102 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:10,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:10,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:10,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:10,106 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:10,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:10,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:10,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,107 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,108 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,108 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:10,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:10,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:10,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:10,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:10,110 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,110 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:10,110 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:10,110 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:10,113 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,113 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,114 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:10,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78087d67[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:10,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:10,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,117 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:10,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:10,117 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:10,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:10,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:10,122 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:10,122 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:10,122 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:10,122 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:10,122 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:10,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,126 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:10,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:10,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:10,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,137 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:10,137 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:10,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861430181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861430183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861430184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,237 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:10,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:10,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:10,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:10,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:10,240 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:10,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:10,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:10,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,242 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:10,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:10,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:10,245 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:10,245 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,245 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:10,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:10,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:10,248 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:10,248 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:10,250 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,250 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14aee894[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:10,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:10,252 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,254 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:10,254 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:10,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,254 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:10,256 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:10,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:10,259 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:10,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:10,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:10,259 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:10,259 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:10,261 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,261 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,261 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,263 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,264 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,266 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,266 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,267 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,267 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,267 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:10,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:10,267 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,268 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,268 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,268 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,267 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,268 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:10,268 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:10,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:10,332 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/13df56c11981450f8908dac1b2ed8387 2023-04-07 09:56:10,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/31e2df401b074f51bacefc80f2932189 2023-04-07 09:56:10,368 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:10,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:10,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:10,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:10,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:10,372 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:10,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:10,372 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:10,373 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,373 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,375 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,375 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,376 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,376 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:10,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:10,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:10,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:10,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:10,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,380 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:10,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:10,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:10,388 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,388 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:10,388 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:10,388 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:10,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:10,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:10,388 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:10,389 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,390 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:10,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791af309[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:10,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:10,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,393 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:10,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:10,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:10,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,395 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:10,395 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:10,397 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:10,397 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:10,397 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:10,397 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:10,397 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:10,399 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,399 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,400 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,400 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,401 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,416 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:10,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:10,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:10,416 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,417 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,416 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,417 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:10,417 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:10,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861430487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861430487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861430487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,517 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:10,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:10,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:10,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:10,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:10,521 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:10,521 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,521 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:10,521 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:10,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,522 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,523 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,523 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,524 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:10,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:10,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:10,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:10,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:10,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:10,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:10,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:10,540 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,540 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:10,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:10,540 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:10,540 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:10,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:10,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:10,540 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:10,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,542 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,542 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:10,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:10,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c1bc27[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:10,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:10,545 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,547 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,548 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:10,548 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:10,548 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:10,548 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:10,550 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:10,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,550 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:10,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:10,554 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:10,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:10,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:10,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:10,554 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:10,555 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,556 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,558 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,573 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,577 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,577 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:10,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:10,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:10,578 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:10,578 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:10,678 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:10,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:10,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:10,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:10,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:10,684 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:10,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:10,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:10,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,694 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,694 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,695 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:10,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:10,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:10,703 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,703 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:10,703 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:10,703 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:10,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,705 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,705 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:10,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f99189d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:10,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:10,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,710 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:10,710 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:10,711 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:10,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,711 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:10,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:10,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,713 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:10,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:10,716 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:10,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:10,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:10,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:10,716 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:10,718 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,718 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:10,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:10,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,721 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:10,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:10,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:10,723 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,724 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:10,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:10,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:10,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:10,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:10,724 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:10,724 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:10,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/29aff34f714747b5b867607521ce6bac 2023-04-07 09:56:10,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/13df56c11981450f8908dac1b2ed8387 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/13df56c11981450f8908dac1b2ed8387 2023-04-07 09:56:10,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/13df56c11981450f8908dac1b2ed8387, entries=250, sequenceid=275, filesize=39.2 K 2023-04-07 09:56:10,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/31e2df401b074f51bacefc80f2932189 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/31e2df401b074f51bacefc80f2932189 2023-04-07 09:56:10,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/31e2df401b074f51bacefc80f2932189, entries=150, sequenceid=275, filesize=11.9 K 2023-04-07 09:56:10,781 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/29aff34f714747b5b867607521ce6bac as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/29aff34f714747b5b867607521ce6bac 2023-04-07 09:56:10,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/29aff34f714747b5b867607521ce6bac, entries=150, sequenceid=275, filesize=11.9 K 2023-04-07 09:56:10,786 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~428.30 KB/438576, currentSize=33.54 KB/34350 for 448acd9b275d13bcb18a1b50f0aa55df in 925ms, sequenceid=275, compaction requested=true 2023-04-07 09:56:10,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:10,786 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:10,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:10,787 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:10,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:10,787 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:10,788 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 129998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:10,788 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/A is initiating minor compaction (all files) 2023-04-07 09:56:10,788 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/A in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,788 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/14a7cc3242174f04a71d58a7a6da3d9c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/40f03b4909bf47cda272b91b3ee31023, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/abe9e3a808344d27889febc59f673d81, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5e3e58b934264c8d9634a1d65a87f918, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/13df56c11981450f8908dac1b2ed8387] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=127.0 K 2023-04-07 09:56:10,788 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55818 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:10,788 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/B is initiating minor compaction (all files) 2023-04-07 09:56:10,788 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/B in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,788 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8a4b8e66de5a4540867c4cf933e652c3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8e42dc7e180d4c529853a24f8a4fec83, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/03eef0fec755435db9cb23053b44c6bb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/a957bb862fd4447d8f55562eb005a0c0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/31e2df401b074f51bacefc80f2932189] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=54.5 K 2023-04-07 09:56:10,789 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8a4b8e66de5a4540867c4cf933e652c3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680861366354 2023-04-07 09:56:10,789 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 14a7cc3242174f04a71d58a7a6da3d9c, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680861366354 2023-04-07 09:56:10,789 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8e42dc7e180d4c529853a24f8a4fec83, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680861367541 2023-04-07 09:56:10,789 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 40f03b4909bf47cda272b91b3ee31023, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680861367541 2023-04-07 09:56:10,789 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 03eef0fec755435db9cb23053b44c6bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680861367653 2023-04-07 09:56:10,790 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting abe9e3a808344d27889febc59f673d81, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680861367653 2023-04-07 09:56:10,790 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a957bb862fd4447d8f55562eb005a0c0, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680861369759 2023-04-07 09:56:10,790 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 31e2df401b074f51bacefc80f2932189, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861369853 2023-04-07 09:56:10,790 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5e3e58b934264c8d9634a1d65a87f918, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680861369759 2023-04-07 09:56:10,791 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 13df56c11981450f8908dac1b2ed8387, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861369852 2023-04-07 09:56:10,799 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#B#compaction#626 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:10,809 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#A#compaction#627 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:10,822 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/9dff6a69043d4aeeb8629185baff2fff as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9dff6a69043d4aeeb8629185baff2fff 2023-04-07 09:56:10,824 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:10,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:10,827 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/B of 448acd9b275d13bcb18a1b50f0aa55df into 9dff6a69043d4aeeb8629185baff2fff(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:10,827 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:10,827 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/B, priority=11, startTime=1680861370787; duration=0sec 2023-04-07 09:56:10,827 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:10,827 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:10,829 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55818 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:10,829 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/C is initiating minor compaction (all files) 2023-04-07 09:56:10,829 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/C in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,829 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4472b330a1ff48c3937db8210c0c6883, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/cb6c464667f44f33ab9a7e6404f1b07e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/fb1dbb49de3744999342d29edf108542, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/68359d9e4a9f44e28c4b0c766619a3c6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/29aff34f714747b5b867607521ce6bac] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=54.5 K 2023-04-07 09:56:10,830 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4472b330a1ff48c3937db8210c0c6883, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1680861366354 2023-04-07 09:56:10,830 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting cb6c464667f44f33ab9a7e6404f1b07e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680861367541 2023-04-07 09:56:10,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:10,831 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting fb1dbb49de3744999342d29edf108542, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680861367653 2023-04-07 09:56:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:10,832 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:10,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:10,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:10,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:10,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:10,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,834 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:10,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:10,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:10,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:10,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:10,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:10,836 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:10,840 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 68359d9e4a9f44e28c4b0c766619a3c6, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680861369759 2023-04-07 09:56:10,840 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 29aff34f714747b5b867607521ce6bac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861369853 2023-04-07 09:56:10,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:10,846 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:10,846 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:10,846 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:10,846 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-07 09:56:10,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:10,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:10,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:10,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:10,853 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:10,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:10,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:10,854 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:10,863 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#C#compaction#628 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:10,925 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/4146d4f46c6041a4b9093d91e6fff989 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4146d4f46c6041a4b9093d91e6fff989 2023-04-07 09:56:10,929 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/C of 448acd9b275d13bcb18a1b50f0aa55df into 4146d4f46c6041a4b9093d91e6fff989(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:10,930 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:10,930 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/C, priority=11, startTime=1680861370787; duration=0sec 2023-04-07 09:56:10,930 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:10,942 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040731f854f4feba4aeebccd9992b95afde2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040731f854f4feba4aeebccd9992b95afde2 2023-04-07 09:56:10,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861370908, max timestamp=1680861370944Num uniques 859; 2023-04-07 09:56:10,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:10,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861370908, max timestamp=1680861370944Num uniques 859; 2023-04-07 09:56:10,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:10,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861370908, max timestamp=1680861370944Num uniques 859; 2023-04-07 09:56:10,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:10,969 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/7fa7783763b34c4c9fd5f565650b24be 2023-04-07 09:56:10,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861430989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861430990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861430992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861430993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:10,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:10,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861430995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861431094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861431094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,258 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/a0182f3ec71543f1a472c4fe56169273 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/a0182f3ec71543f1a472c4fe56169273 2023-04-07 09:56:11,263 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/A of 448acd9b275d13bcb18a1b50f0aa55df into a0182f3ec71543f1a472c4fe56169273(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:11,263 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:11,263 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/A, priority=11, startTime=1680861370786; duration=0sec 2023-04-07 09:56:11,263 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:11,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861431298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861431299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,395 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/4e162f920ff34a38b4a08782f9c7b59f 2023-04-07 09:56:11,413 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/8cc4ec66894644c8bbea2085b2523444 2023-04-07 09:56:11,417 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/7fa7783763b34c4c9fd5f565650b24be as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/7fa7783763b34c4c9fd5f565650b24be 2023-04-07 09:56:11,422 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/7fa7783763b34c4c9fd5f565650b24be, entries=200, sequenceid=286, filesize=32.3 K 2023-04-07 09:56:11,423 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/4e162f920ff34a38b4a08782f9c7b59f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4e162f920ff34a38b4a08782f9c7b59f 2023-04-07 09:56:11,427 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4e162f920ff34a38b4a08782f9c7b59f, entries=150, sequenceid=286, filesize=11.9 K 2023-04-07 09:56:11,428 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/8cc4ec66894644c8bbea2085b2523444 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8cc4ec66894644c8bbea2085b2523444 2023-04-07 09:56:11,433 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8cc4ec66894644c8bbea2085b2523444, entries=150, sequenceid=286, filesize=11.9 K 2023-04-07 09:56:11,434 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 448acd9b275d13bcb18a1b50f0aa55df in 588ms, sequenceid=286, compaction requested=false 2023-04-07 09:56:11,434 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:11,434 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:11,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:11,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:11,434 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,435 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:11,435 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:11,438 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,438 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:11,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,439 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c90afb2[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:11,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:11,441 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,448 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:11,448 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,448 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:11,449 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:11,452 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:11,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:11,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:11,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:11,452 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:11,453 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,453 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,453 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:11,454 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:11,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:11,455 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:11,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:11,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:11,457 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:11,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:11,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:11,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:11,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:11,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,457 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:11,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,458 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:11,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:11,458 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:11,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:11,558 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:11,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:11,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:11,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:11,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:11,562 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:11,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:11,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:11,562 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:11,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,563 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:11,565 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:11,565 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:11,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:11,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:11,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:11,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:11,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:11,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:11,580 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:11,580 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:11,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:11,582 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 09:56:11,582 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:11,585 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:11,585 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:11,585 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:11,585 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:11,585 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:11,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:11,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:11,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861431614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38334 deadline: 1680861431615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861431717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861431719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861431921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:11,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:11,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861431923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861431999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,001 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407385675f42e99486fa00067a8fdc6bdb9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407385675f42e99486fa00067a8fdc6bdb9 2023-04-07 09:56:12,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861432003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861432004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38308 deadline: 1680861432225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861432227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,407 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=325, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/fb691cfe282b472c9a8eadb85196c908 2023-04-07 09:56:12,451 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/0da994fe2b8e444ab910c9f66fc34e9f 2023-04-07 09:56:12,462 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/478596edd77a4079b49f5e6f0e3b3892 2023-04-07 09:56:12,466 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/fb691cfe282b472c9a8eadb85196c908 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/fb691cfe282b472c9a8eadb85196c908 2023-04-07 09:56:12,470 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/fb691cfe282b472c9a8eadb85196c908, entries=150, sequenceid=325, filesize=25.4 K 2023-04-07 09:56:12,470 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/0da994fe2b8e444ab910c9f66fc34e9f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0da994fe2b8e444ab910c9f66fc34e9f 2023-04-07 09:56:12,475 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0da994fe2b8e444ab910c9f66fc34e9f, entries=150, sequenceid=325, filesize=11.9 K 2023-04-07 09:56:12,476 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/478596edd77a4079b49f5e6f0e3b3892 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/478596edd77a4079b49f5e6f0e3b3892 2023-04-07 09:56:12,479 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/478596edd77a4079b49f5e6f0e3b3892, entries=150, sequenceid=325, filesize=11.9 K 2023-04-07 09:56:12,480 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 448acd9b275d13bcb18a1b50f0aa55df in 898ms, sequenceid=325, compaction requested=true 2023-04-07 09:56:12,480 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:12,480 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:12,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:12,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:12,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:12,480 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:12,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,483 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:12,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:12,484 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:12,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623c9f92[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:12,485 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:12,486 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,487 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:12,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,488 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:12,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:12,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:12,490 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,490 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:12,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:12,492 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:12,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:12,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:12,493 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:12,493 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:12,495 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,495 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,495 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:12,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,496 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,499 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,499 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:12,499 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:12,499 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:12,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:12,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:12,499 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:12,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:12,499 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:12,500 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:12,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,500 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:12,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:12,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:12,500 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,600 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:12,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:12,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:12,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:12,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:12,604 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:12,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:12,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:12,604 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:12,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,605 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:12,606 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:12,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:12,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:12,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:12,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:12,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:12,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:12,609 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:12,609 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:12,609 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:12,623 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040718fc1202a03d4bbaa03fd3f2c43dfef1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040718fc1202a03d4bbaa03fd3f2c43dfef1 2023-04-07 09:56:12,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:12,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:12,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861372762, max timestamp=1680861372772Num uniques 859; 2023-04-07 09:56:12,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:12,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861372762, max timestamp=1680861372772Num uniques 859; 2023-04-07 09:56:12,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:12,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861372762, max timestamp=1680861372772Num uniques 859; 2023-04-07 09:56:12,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:12,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861432807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861432811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861432911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:12,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:12,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861432914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,037 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/ff9b17c8e77444b2a3df4434404b5129 2023-04-07 09:56:13,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:13,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861433114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:13,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861433116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:13,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861433417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:13,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861433419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,446 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/d82506d46a9d4b24bccfd58ba146b077 2023-04-07 09:56:13,857 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/1687d48eea7847c7bfbd52174623f67b 2023-04-07 09:56:13,860 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/ff9b17c8e77444b2a3df4434404b5129 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff9b17c8e77444b2a3df4434404b5129 2023-04-07 09:56:13,865 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff9b17c8e77444b2a3df4434404b5129, entries=100, sequenceid=335, filesize=18.6 K 2023-04-07 09:56:13,865 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/d82506d46a9d4b24bccfd58ba146b077 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d82506d46a9d4b24bccfd58ba146b077 2023-04-07 09:56:13,868 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d82506d46a9d4b24bccfd58ba146b077, entries=100, sequenceid=335, filesize=9.5 K 2023-04-07 09:56:13,869 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/1687d48eea7847c7bfbd52174623f67b as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1687d48eea7847c7bfbd52174623f67b 2023-04-07 09:56:13,872 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1687d48eea7847c7bfbd52174623f67b, entries=100, sequenceid=335, filesize=9.5 K 2023-04-07 09:56:13,873 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 448acd9b275d13bcb18a1b50f0aa55df in 1264ms, sequenceid=335, compaction requested=true 2023-04-07 09:56:13,873 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:13,873 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:13,873 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:13,873 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:13,873 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,873 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:13,873 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:13,879 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,879 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:13,879 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,879 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b053776[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:13,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:13,880 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,881 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:13,881 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:13,881 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:13,882 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,882 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:13,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:13,884 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:13,884 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:13,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:13,886 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:13,886 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:13,886 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:13,886 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:13,886 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:13,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:13,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:13,888 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:13,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:13,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:13,889 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:13,890 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,891 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,891 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:13,892 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:13,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:13,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:13,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:13,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:13,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:13,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:13,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:13,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:13,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:13,892 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:13,892 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:13,892 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:13,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-07 09:56:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:56:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:56:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:13,925 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-07 09:56:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:13,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:13,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861433937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861433941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:13,970 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407287f76718cb24a6794063e64fef4b68c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407287f76718cb24a6794063e64fef4b68c 2023-04-07 09:56:13,992 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:13,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:13,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,016 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=373, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/8c4a7f64b5304b6997e9578015e91756 2023-04-07 09:56:14,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861434014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,024 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,024 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,025 DEBUG [Thread-4400] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:56:14,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861434020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861434021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,027 DEBUG [Thread-4392] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:56:14,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,028 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,029 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,030 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,031 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,032 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,032 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,032 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,032 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,032 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,033 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,034 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,033 DEBUG [Thread-4394] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., hostname=jenkins-hbase4.apache.org,35881,1680861218288, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-07 09:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,037 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c3f957b[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,038 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,040 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,040 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,040 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,042 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,042 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861434042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,045 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,046 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,046 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,047 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861434045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,051 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,052 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,052 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,056 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:14,056 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:14,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,057 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:14,057 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:14,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:14,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,157 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:14,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:14,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,162 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,162 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,166 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,167 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,172 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,172 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,173 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,176 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,177 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551b4471[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,177 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,178 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,178 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,179 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,181 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:14,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,185 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,185 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,185 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,188 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,189 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,191 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:14,194 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:14,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:14,194 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:14,194 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:14,196 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,196 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861434246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861434253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,294 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:14,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:14,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,299 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,299 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,299 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,299 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,300 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,300 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,301 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,301 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,303 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,303 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,304 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,304 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,304 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,304 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,307 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,307 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@535d914f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,312 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,312 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,313 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,313 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,317 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,317 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:14,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,320 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,320 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,320 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,330 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,333 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,335 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,337 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:14,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:14,338 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:14,338 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:14,337 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,338 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/26e12a550d1147e7acf8d9d1cb0101ff 2023-04-07 09:56:14,438 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:14,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:14,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,443 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,443 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,446 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,456 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,457 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,457 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,457 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,458 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/a5126f5ac4ac4e78846238dfe4d76661 2023-04-07 09:56:14,461 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/8c4a7f64b5304b6997e9578015e91756 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8c4a7f64b5304b6997e9578015e91756 2023-04-07 09:56:14,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b1a2d5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,463 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8c4a7f64b5304b6997e9578015e91756, entries=200, sequenceid=373, filesize=32.3 K 2023-04-07 09:56:14,466 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,466 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,467 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/26e12a550d1147e7acf8d9d1cb0101ff as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/26e12a550d1147e7acf8d9d1cb0101ff 2023-04-07 09:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,467 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,469 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,469 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/26e12a550d1147e7acf8d9d1cb0101ff, entries=150, sequenceid=373, filesize=11.9 K 2023-04-07 09:56:14,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/a5126f5ac4ac4e78846238dfe4d76661 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a5126f5ac4ac4e78846238dfe4d76661 2023-04-07 09:56:14,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:14,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,474 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,474 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,474 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,476 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,477 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,478 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a5126f5ac4ac4e78846238dfe4d76661, entries=150, sequenceid=373, filesize=11.9 K 2023-04-07 09:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,480 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 448acd9b275d13bcb18a1b50f0aa55df in 556ms, sequenceid=373, compaction requested=true 2023-04-07 09:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,480 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:14,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:14,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:14,481 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:14,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:14,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:14,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,482 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:14,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,482 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:14,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,482 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137706 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:14,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,482 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/A is initiating minor compaction (all files) 2023-04-07 09:56:14,482 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,483 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,483 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/A in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,483 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/a0182f3ec71543f1a472c4fe56169273, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/7fa7783763b34c4c9fd5f565650b24be, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/fb691cfe282b472c9a8eadb85196c908, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff9b17c8e77444b2a3df4434404b5129, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8c4a7f64b5304b6997e9578015e91756] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=134.5 K 2023-04-07 09:56:14,483 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58930 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:14,483 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting a0182f3ec71543f1a472c4fe56169273, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861369853 2023-04-07 09:56:14,483 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/B is initiating minor compaction (all files) 2023-04-07 09:56:14,483 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/B in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,483 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9dff6a69043d4aeeb8629185baff2fff, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4e162f920ff34a38b4a08782f9c7b59f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0da994fe2b8e444ab910c9f66fc34e9f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d82506d46a9d4b24bccfd58ba146b077, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/26e12a550d1147e7acf8d9d1cb0101ff] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=57.5 K 2023-04-07 09:56:14,483 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 7fa7783763b34c4c9fd5f565650b24be, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680861369874 2023-04-07 09:56:14,484 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9dff6a69043d4aeeb8629185baff2fff, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861369853 2023-04-07 09:56:14,484 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting fb691cfe282b472c9a8eadb85196c908, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680861370978 2023-04-07 09:56:14,484 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ff9b17c8e77444b2a3df4434404b5129, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680861371612 2023-04-07 09:56:14,484 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4e162f920ff34a38b4a08782f9c7b59f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680861369876 2023-04-07 09:56:14,485 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 8c4a7f64b5304b6997e9578015e91756, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680861372796 2023-04-07 09:56:14,485 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0da994fe2b8e444ab910c9f66fc34e9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680861370978 2023-04-07 09:56:14,485 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting d82506d46a9d4b24bccfd58ba146b077, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680861371612 2023-04-07 09:56:14,486 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 26e12a550d1147e7acf8d9d1cb0101ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680861372796 2023-04-07 09:56:14,496 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#B#compaction#642 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:14,499 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#A#compaction#641 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:14,513 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/9a1b7f6e81c449afbfb29ecc0c5bb35d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9a1b7f6e81c449afbfb29ecc0c5bb35d 2023-04-07 09:56:14,516 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/B of 448acd9b275d13bcb18a1b50f0aa55df into 9a1b7f6e81c449afbfb29ecc0c5bb35d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:14,516 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:14,516 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/B, priority=11, startTime=1680861374481; duration=0sec 2023-04-07 09:56:14,516 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:14,516 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-07 09:56:14,518 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58930 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-07 09:56:14,518 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/C is initiating minor compaction (all files) 2023-04-07 09:56:14,518 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/C in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,518 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4146d4f46c6041a4b9093d91e6fff989, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8cc4ec66894644c8bbea2085b2523444, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/478596edd77a4079b49f5e6f0e3b3892, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1687d48eea7847c7bfbd52174623f67b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a5126f5ac4ac4e78846238dfe4d76661] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=57.5 K 2023-04-07 09:56:14,519 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 4146d4f46c6041a4b9093d91e6fff989, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680861369853 2023-04-07 09:56:14,520 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8cc4ec66894644c8bbea2085b2523444, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680861369876 2023-04-07 09:56:14,520 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 478596edd77a4079b49f5e6f0e3b3892, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680861370978 2023-04-07 09:56:14,521 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1687d48eea7847c7bfbd52174623f67b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680861371612 2023-04-07 09:56:14,522 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a5126f5ac4ac4e78846238dfe4d76661, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680861372796 2023-04-07 09:56:14,531 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#C#compaction#643 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:14,538 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/0c5773cbae3c4264832133894317e776 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/0c5773cbae3c4264832133894317e776 2023-04-07 09:56:14,540 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/b773abbc8a11431d953ef669da263d3a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b773abbc8a11431d953ef669da263d3a 2023-04-07 09:56:14,542 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/A of 448acd9b275d13bcb18a1b50f0aa55df into 0c5773cbae3c4264832133894317e776(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:14,542 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:14,542 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/A, priority=11, startTime=1680861374480; duration=0sec 2023-04-07 09:56:14,542 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:14,544 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/C of 448acd9b275d13bcb18a1b50f0aa55df into b773abbc8a11431d953ef669da263d3a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:14,544 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:14,544 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/C, priority=11, startTime=1680861374481; duration=0sec 2023-04-07 09:56:14,545 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:14,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:14,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:56:14,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:14,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:14,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:14,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:14,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:14,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:14,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861374567, max timestamp=1680861374579Num uniques 859; 2023-04-07 09:56:14,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861374567, max timestamp=1680861374579Num uniques 859; 2023-04-07 09:56:14,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:56:14,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861374567, max timestamp=1680861374579Num uniques 859; 2023-04-07 09:56:14,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:56:14,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:56:14,582 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:14,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:14,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,586 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,586 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,586 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,586 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,586 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,586 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,588 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,588 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,590 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,590 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,590 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,590 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,591 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,591 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,591 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,591 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,591 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,591 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,593 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,593 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,594 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,594 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61c57bd5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,597 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,599 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,599 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,599 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,600 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,602 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,602 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,602 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,602 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,605 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,606 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,607 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,609 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,612 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:14,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:14,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,612 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:14,612 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,612 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,613 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861434613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861434614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,712 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:14,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:14,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,716 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,716 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,717 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,719 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861434718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,719 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861434718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,724 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,725 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,726 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,726 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,726 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,726 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,726 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,729 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,729 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,730 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ef29350[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,731 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,731 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,733 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,733 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,733 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,733 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,734 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,734 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,735 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:14,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,738 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,738 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,738 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,738 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,738 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,744 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,745 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,745 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,747 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,753 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:14,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,754 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:14,754 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:14,754 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:14,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,753 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,854 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:14,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:14,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,857 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,857 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,859 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,859 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,860 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,860 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,860 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,860 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,860 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,862 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,862 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519ab028[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,863 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,865 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,865 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,866 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:14,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,869 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,870 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,870 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,870 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,870 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,871 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,871 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,873 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,875 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:14,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:14,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:14,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,875 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:14,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,875 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:14,875 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,876 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,876 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,876 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861434920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:14,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861434921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,962 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e60f2bb47c31445d91cb2d4e2b97d68b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e60f2bb47c31445d91cb2d4e2b97d68b 2023-04-07 09:56:14,976 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:14,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:14,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:14,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:14,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:14,979 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:14,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:14,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:14,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,982 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:14,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:14,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,983 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:14,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:14,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:14,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:14,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:14,985 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,985 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:14,985 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:14,985 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:14,985 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:14,986 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,986 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,987 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:14,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db117c4[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:14,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:14,988 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:14,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,990 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:14,991 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:14,995 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:14,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:14,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:14,995 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:14,995 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:14,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,998 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:14,998 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:14,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,000 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,002 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,002 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:15,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:15,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:15,003 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:15,003 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:15,103 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:15,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:15,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:15,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:15,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:15,109 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:15,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:15,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:15,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,109 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:15,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:15,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:15,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:15,120 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:15,120 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:15,120 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:15,120 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:15,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:15,122 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,122 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:15,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1760c474[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:15,124 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:15,125 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,126 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:15,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:15,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:15,127 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:15,128 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:15,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:15,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:15,135 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:15,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:15,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:15,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:15,135 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:15,137 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,138 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,138 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:15,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,139 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,140 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,143 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:15,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:15,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:15,143 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:15,143 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,143 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:15,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861435222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:15,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861435225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,243 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:15,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:15,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:15,248 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:15,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:15,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:15,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,248 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,250 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,250 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:15,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:15,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:15,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:15,252 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:15,252 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,252 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:15,258 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,258 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,258 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:15,268 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:15,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d72bddb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:15,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:15,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,271 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:15,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:15,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:15,273 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:15,273 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:15,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:15,276 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:15,276 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:15,276 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:15,276 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:15,276 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:15,278 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:15,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,279 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,281 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,282 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,284 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,284 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,284 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:15,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:15,284 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:15,284 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:15,285 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:15,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,284 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,285 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:15,285 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,285 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,285 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,285 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:15,370 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=386, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/5dd2c4f6ec984570988e580618197d0d 2023-04-07 09:56:15,385 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:15,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:15,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/1127d55a3ba54150871f8e134a7ac778 2023-04-07 09:56:15,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:15,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:15,392 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:15,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:15,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:15,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,392 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,394 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,394 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:15,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:15,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:15,404 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:15,404 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:15,404 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:15,405 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:15,405 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:15,407 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,407 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,407 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,407 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ed5bad[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:15,409 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:15,410 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,411 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,411 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:15,411 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:15,411 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:15,411 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,411 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:15,413 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/02b46161d7f14ce0919ef581a27682fb 2023-04-07 09:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:15,417 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:15,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:15,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:15,417 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:15,417 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:15,419 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:15,420 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:15,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,420 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:15,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,421 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:15,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/5dd2c4f6ec984570988e580618197d0d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5dd2c4f6ec984570988e580618197d0d 2023-04-07 09:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:15,424 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5dd2c4f6ec984570988e580618197d0d, entries=200, sequenceid=386, filesize=32.3 K 2023-04-07 09:56:15,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/1127d55a3ba54150871f8e134a7ac778 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1127d55a3ba54150871f8e134a7ac778 2023-04-07 09:56:15,428 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,428 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:15,428 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:15,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:15,428 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,428 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,429 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,429 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,429 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:15,429 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,428 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:15,429 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:15,429 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:15,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:15,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1127d55a3ba54150871f8e134a7ac778, entries=150, sequenceid=386, filesize=11.9 K 2023-04-07 09:56:15,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/02b46161d7f14ce0919ef581a27682fb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/02b46161d7f14ce0919ef581a27682fb 2023-04-07 09:56:15,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/02b46161d7f14ce0919ef581a27682fb, entries=150, sequenceid=386, filesize=11.9 K 2023-04-07 09:56:15,440 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 448acd9b275d13bcb18a1b50f0aa55df in 888ms, sequenceid=386, compaction requested=false 2023-04-07 09:56:15,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:15,529 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:15,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:15,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:15,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:15,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:15,532 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:15,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:15,532 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:15,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,533 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,534 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:15,534 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:15,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,535 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:15,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:15,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:15,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:15,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:15,537 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:15,537 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:15,554 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040709c5e5d8b57d40989ccc94b1c066a407 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040709c5e5d8b57d40989ccc94b1c066a407 2023-04-07 09:56:15,562 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=422, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/b4d1221159d8451c998bc8776e20de69 2023-04-07 09:56:15,574 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/365f8e6577db423698f2fb9409d7c3f9 2023-04-07 09:56:15,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:15,732 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:15,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:15,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861435741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861435742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:15,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861435845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:15,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861435845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:15,988 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/00fa8c9d68e84e7a81f5e9a9c7e6551a 2023-04-07 09:56:16,000 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/b4d1221159d8451c998bc8776e20de69 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b4d1221159d8451c998bc8776e20de69 2023-04-07 09:56:16,005 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b4d1221159d8451c998bc8776e20de69, entries=150, sequenceid=422, filesize=25.4 K 2023-04-07 09:56:16,006 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/365f8e6577db423698f2fb9409d7c3f9 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/365f8e6577db423698f2fb9409d7c3f9 2023-04-07 09:56:16,011 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/365f8e6577db423698f2fb9409d7c3f9, entries=150, sequenceid=422, filesize=11.9 K 2023-04-07 09:56:16,011 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/00fa8c9d68e84e7a81f5e9a9c7e6551a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/00fa8c9d68e84e7a81f5e9a9c7e6551a 2023-04-07 09:56:16,017 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/00fa8c9d68e84e7a81f5e9a9c7e6551a, entries=150, sequenceid=422, filesize=11.9 K 2023-04-07 09:56:16,018 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 448acd9b275d13bcb18a1b50f0aa55df in 481ms, sequenceid=422, compaction requested=true 2023-04-07 09:56:16,019 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:16,019 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,019 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,021 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,021 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,021 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,021 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb77e63[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,023 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,024 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,025 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,029 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,032 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,032 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,032 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,034 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,034 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,034 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,037 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,042 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,049 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,049 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,049 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,049 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,050 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,050 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,050 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:16,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-07 09:56:16,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:16,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:16,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:16,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:16,051 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:16,051 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:16,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861436094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861436098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,150 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:16,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:16,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:16,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:16,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:16,160 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:16,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:16,160 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:16,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,161 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,165 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,165 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:16,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:16,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:16,168 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:16,169 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,169 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:16,169 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:16,169 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,171 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,171 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de54ee5[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,172 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,174 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,174 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,174 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,174 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,174 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,175 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,175 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,178 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,178 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,178 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,181 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,181 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,189 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,193 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,193 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,194 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,194 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,194 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:16,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861436198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861436201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,294 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:16,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:16,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:16,303 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:16,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:16,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:16,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,303 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,304 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:16,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:16,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:16,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:16,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:16,306 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,306 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,308 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5581a2e7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,310 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,311 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,311 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,311 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,313 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,313 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,317 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,317 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,318 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,318 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,318 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,319 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,320 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,320 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,322 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,328 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,331 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,332 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:16,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,332 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,333 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,333 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861436402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861436404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,433 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:16,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:16,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:16,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:16,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:16,436 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:16,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:16,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:16,437 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,437 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,439 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,439 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:16,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:16,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:16,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:16,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,440 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:16,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:16,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:16,441 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,441 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:16,441 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:16,441 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,441 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,442 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,442 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,443 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5632abc9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,444 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,444 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,445 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,446 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,446 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,447 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,447 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,449 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,449 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,449 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,452 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,452 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,452 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,452 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,453 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,456 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:16,456 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,456 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,456 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,472 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407b6bf06f88d244aadb8020b6a7a6d43ec to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b6bf06f88d244aadb8020b6a7a6d43ec 2023-04-07 09:56:16,556 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:16,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:16,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:16,559 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:16,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:16,559 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:16,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,560 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,561 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,561 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:16,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:16,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:16,562 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:16,562 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,562 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:16,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:16,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:16,563 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,563 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:16,563 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:16,563 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,563 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,568 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5171930a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,570 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,571 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,571 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,571 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,572 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,574 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,574 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,576 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,577 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,581 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,580 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:16,581 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,581 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,581 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,581 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,581 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,681 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:16,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:16,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:16,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:16,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:16,684 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:16,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:16,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:16,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,686 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:16,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:16,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:16,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:16,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:16,687 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,687 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:16,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:16,687 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:16,687 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,687 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,689 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56647da7[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,691 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,692 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,692 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,692 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,694 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,694 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,697 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,697 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,697 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,700 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,701 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,703 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,704 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,704 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,704 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,704 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861436707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:16,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861436708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,804 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:16,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:16,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:16,808 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:16,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:16,808 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:16,809 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,809 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,810 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,810 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,811 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:16,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:16,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:16,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:16,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:16,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,817 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:16,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:16,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:16,817 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,818 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,819 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,819 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eaa7b16[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,828 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,829 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,830 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,830 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,832 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,832 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,835 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,835 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,837 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,837 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,839 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,841 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,841 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,841 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,841 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:16,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,876 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=435, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/90db6084db2847cc9752c01207e1622e 2023-04-07 09:56:16,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/8735ef5cb1804340bb625e703a8ea348 2023-04-07 09:56:16,942 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:16,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:16,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:16,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:16,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:16,945 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:16,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:16,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:16,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,945 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,946 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,946 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,947 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:16,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:16,948 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,948 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:16,948 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:16,948 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:16,951 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d85f5e9[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:16,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:16,953 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,957 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:16,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:16,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:16,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,957 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:16,958 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:16,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,958 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:16,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:16,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:16,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:16,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:16,961 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:16,961 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:16,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:16,963 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:16,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,963 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,965 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:16,968 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:16,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:16,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:16,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:16,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:16,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:16,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:16,968 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:16,968 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:16,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:16,969 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:16,969 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,069 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:17,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:17,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:17,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:17,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:17,078 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:17,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:17,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:17,078 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:17,080 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,080 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,081 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:17,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:17,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,081 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,082 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:17,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:17,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:17,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:17,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:17,083 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:17,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:17,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:17,096 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:17,096 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,099 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1031d690[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:17,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:17,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,103 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:17,103 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:17,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:17,106 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:17,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:17,110 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:17,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:17,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:17,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:17,111 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:17,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:17,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,112 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:17,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:17,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:17,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:17,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,119 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:17,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:17,120 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:17,120 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:17,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:17,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861437213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:17,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861437214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,220 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:17,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:17,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:17,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:17,224 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:17,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:17,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:17,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:17,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,224 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,225 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:17,225 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:17,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,226 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:17,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:17,227 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,227 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:17,227 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:17,227 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:17,228 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:17,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:17,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cd1874[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:17,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:17,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:17,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:17,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:17,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:17,236 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:17,236 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:17,240 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:17,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:17,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:17,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:17,240 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:17,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,244 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,245 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:17,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:17,247 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:17,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:17,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:17,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,247 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:17,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,247 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:17,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,247 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,248 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:17,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/e49e1628d4a941e3afa98034955ca944 2023-04-07 09:56:17,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/90db6084db2847cc9752c01207e1622e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/90db6084db2847cc9752c01207e1622e 2023-04-07 09:56:17,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/90db6084db2847cc9752c01207e1622e, entries=200, sequenceid=435, filesize=32.3 K 2023-04-07 09:56:17,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/8735ef5cb1804340bb625e703a8ea348 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8735ef5cb1804340bb625e703a8ea348 2023-04-07 09:56:17,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8735ef5cb1804340bb625e703a8ea348, entries=150, sequenceid=435, filesize=11.9 K 2023-04-07 09:56:17,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/e49e1628d4a941e3afa98034955ca944 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/e49e1628d4a941e3afa98034955ca944 2023-04-07 09:56:17,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/e49e1628d4a941e3afa98034955ca944, entries=150, sequenceid=435, filesize=11.9 K 2023-04-07 09:56:17,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 448acd9b275d13bcb18a1b50f0aa55df in 1266ms, sequenceid=435, compaction requested=true 2023-04-07 09:56:17,317 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:17,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:17,317 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:56:17,317 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:56:17,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:17,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:17,318 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118878 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:56:17,318 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:56:17,318 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/A is initiating minor compaction (all files) 2023-04-07 09:56:17,318 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/B is initiating minor compaction (all files) 2023-04-07 09:56:17,318 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/A in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,318 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/B in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,318 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/0c5773cbae3c4264832133894317e776, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5dd2c4f6ec984570988e580618197d0d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b4d1221159d8451c998bc8776e20de69, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/90db6084db2847cc9752c01207e1622e] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=116.1 K 2023-04-07 09:56:17,318 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9a1b7f6e81c449afbfb29ecc0c5bb35d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1127d55a3ba54150871f8e134a7ac778, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/365f8e6577db423698f2fb9409d7c3f9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8735ef5cb1804340bb625e703a8ea348] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=48.2 K 2023-04-07 09:56:17,319 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 0c5773cbae3c4264832133894317e776, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680861372796 2023-04-07 09:56:17,319 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 9a1b7f6e81c449afbfb29ecc0c5bb35d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680861372796 2023-04-07 09:56:17,319 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 5dd2c4f6ec984570988e580618197d0d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680861373929 2023-04-07 09:56:17,320 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b4d1221159d8451c998bc8776e20de69, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680861374607 2023-04-07 09:56:17,320 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 1127d55a3ba54150871f8e134a7ac778, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680861373935 2023-04-07 09:56:17,321 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 90db6084db2847cc9752c01207e1622e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680861375738 2023-04-07 09:56:17,321 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 365f8e6577db423698f2fb9409d7c3f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680861374607 2023-04-07 09:56:17,322 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8735ef5cb1804340bb625e703a8ea348, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680861375740 2023-04-07 09:56:17,338 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#B#compaction#654 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:17,339 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#A#compaction#653 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:17,348 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:17,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:17,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:17,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:17,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:17,365 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:17,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:17,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:17,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:17,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,365 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,369 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:17,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,370 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:17,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:17,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:17,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:17,412 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:17,412 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,412 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:17,412 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-07 09:56:17,412 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:17,413 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:17,413 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:17,413 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:17,413 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:17,413 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:17,477 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/33aa594f8aa94829a238fe160fd53b90 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/33aa594f8aa94829a238fe160fd53b90 2023-04-07 09:56:17,481 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/A of 448acd9b275d13bcb18a1b50f0aa55df into 33aa594f8aa94829a238fe160fd53b90(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:17,481 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:17,481 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/A, priority=12, startTime=1680861377317; duration=0sec 2023-04-07 09:56:17,481 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:17,481 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:56:17,483 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:56:17,483 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/C is initiating minor compaction (all files) 2023-04-07 09:56:17,483 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/C in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,483 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b773abbc8a11431d953ef669da263d3a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/02b46161d7f14ce0919ef581a27682fb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/00fa8c9d68e84e7a81f5e9a9c7e6551a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/e49e1628d4a941e3afa98034955ca944] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=48.2 K 2023-04-07 09:56:17,483 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b773abbc8a11431d953ef669da263d3a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680861372796 2023-04-07 09:56:17,484 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 02b46161d7f14ce0919ef581a27682fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680861373935 2023-04-07 09:56:17,484 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 00fa8c9d68e84e7a81f5e9a9c7e6551a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1680861374607 2023-04-07 09:56:17,484 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting e49e1628d4a941e3afa98034955ca944, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680861375740 2023-04-07 09:56:17,491 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#C#compaction#656 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:17,504 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/6d282e95f61a45b1ae15e0742b273f17 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/6d282e95f61a45b1ae15e0742b273f17 2023-04-07 09:56:17,509 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/C of 448acd9b275d13bcb18a1b50f0aa55df into 6d282e95f61a45b1ae15e0742b273f17(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:17,509 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:17,509 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/C, priority=12, startTime=1680861377317; duration=0sec 2023-04-07 09:56:17,510 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:17,797 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/c13442f022a24fa5bb51d0dbd6d956a1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/c13442f022a24fa5bb51d0dbd6d956a1 2023-04-07 09:56:17,808 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/B of 448acd9b275d13bcb18a1b50f0aa55df into c13442f022a24fa5bb51d0dbd6d956a1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:17,808 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:17,809 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/B, priority=12, startTime=1680861377317; duration=0sec 2023-04-07 09:56:17,809 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:17,877 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407fb5a5c61f0794a29a6803f34598ea59d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fb5a5c61f0794a29a6803f34598ea59d 2023-04-07 09:56:17,894 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/b6ee364cfeb342619227fbe795f1cff1 2023-04-07 09:56:17,913 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/ee0c8a8da7f84e8688033a240d1cfa55 2023-04-07 09:56:17,932 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/09d9cd7b5daf42a781f1214102b93b28 2023-04-07 09:56:17,937 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/b6ee364cfeb342619227fbe795f1cff1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b6ee364cfeb342619227fbe795f1cff1 2023-04-07 09:56:17,940 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b6ee364cfeb342619227fbe795f1cff1, entries=150, sequenceid=454, filesize=25.4 K 2023-04-07 09:56:17,941 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/ee0c8a8da7f84e8688033a240d1cfa55 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ee0c8a8da7f84e8688033a240d1cfa55 2023-04-07 09:56:17,957 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ee0c8a8da7f84e8688033a240d1cfa55, entries=150, sequenceid=454, filesize=11.9 K 2023-04-07 09:56:17,960 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/09d9cd7b5daf42a781f1214102b93b28 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/09d9cd7b5daf42a781f1214102b93b28 2023-04-07 09:56:17,974 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/09d9cd7b5daf42a781f1214102b93b28, entries=150, sequenceid=454, filesize=11.9 K 2023-04-07 09:56:17,975 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 448acd9b275d13bcb18a1b50f0aa55df in 563ms, sequenceid=454, compaction requested=false 2023-04-07 09:56:17,975 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:17,975 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:17,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:17,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:17,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:17,975 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:17,977 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:17,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e2ad10[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:17,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:17,979 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:17,981 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:17,981 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:17,981 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:17,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,981 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:17,989 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:17,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:17,989 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:17,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:17,993 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:17,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:17,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:17,993 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:17,993 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,005 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,006 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,007 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,011 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:18,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,011 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,012 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,012 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:18,012 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:18,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:18,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-07 09:56:18,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:18,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:18,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:18,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:18,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:18,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:18,103 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407dee44c3b3b91440cae74a174679ec291 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407dee44c3b3b91440cae74a174679ec291 2023-04-07 09:56:18,112 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:18,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:18,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:18,116 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:18,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:18,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:18,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,117 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,117 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=468, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/45e4414a41d040eb9ee64a0291e7f39f 2023-04-07 09:56:18,118 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,118 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,119 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:18,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:18,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:18,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:18,120 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:18,121 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:18,123 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,124 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:18,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aaa3d58[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:18,126 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:18,126 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,128 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:18,128 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:18,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861378104, max timestamp=1680861378127Num uniques 859; 2023-04-07 09:56:18,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861378104, max timestamp=1680861378127Num uniques 859; 2023-04-07 09:56:18,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:18,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861378104, max timestamp=1680861378127Num uniques 859; 2023-04-07 09:56:18,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:18,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,130 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:18,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:18,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:18,131 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:18,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:18,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:18,131 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:18,131 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,133 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,133 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,133 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,135 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,140 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/80ecd05d750d409aba071bfb1f9362e7 2023-04-07 09:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,151 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:18,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,151 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,151 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:18,151 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,152 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,152 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/a1c96113a1ba45c382b0d5d612c63142 2023-04-07 09:56:18,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38310 deadline: 1680861438174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38350 deadline: 1680861438175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861438178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/45e4414a41d040eb9ee64a0291e7f39f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/45e4414a41d040eb9ee64a0291e7f39f 2023-04-07 09:56:18,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/45e4414a41d040eb9ee64a0291e7f39f, entries=400, sequenceid=468, filesize=59.8 K 2023-04-07 09:56:18,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/80ecd05d750d409aba071bfb1f9362e7 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/80ecd05d750d409aba071bfb1f9362e7 2023-04-07 09:56:18,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/80ecd05d750d409aba071bfb1f9362e7, entries=150, sequenceid=468, filesize=11.9 K 2023-04-07 09:56:18,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/a1c96113a1ba45c382b0d5d612c63142 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a1c96113a1ba45c382b0d5d612c63142 2023-04-07 09:56:18,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a1c96113a1ba45c382b0d5d612c63142, entries=150, sequenceid=468, filesize=11.9 K 2023-04-07 09:56:18,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 448acd9b275d13bcb18a1b50f0aa55df in 111ms, sequenceid=468, compaction requested=true 2023-04-07 09:56:18,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:18,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:18,201 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:56:18,201 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:56:18,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:18,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:18,203 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:56:18,203 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/B is initiating minor compaction (all files) 2023-04-07 09:56:18,203 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/B in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,203 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/c13442f022a24fa5bb51d0dbd6d956a1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ee0c8a8da7f84e8688033a240d1cfa55, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/80ecd05d750d409aba071bfb1f9362e7] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=36.4 K 2023-04-07 09:56:18,204 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 114096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:56:18,204 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting c13442f022a24fa5bb51d0dbd6d956a1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680861375740 2023-04-07 09:56:18,204 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/A is initiating minor compaction (all files) 2023-04-07 09:56:18,204 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/A in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,204 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/33aa594f8aa94829a238fe160fd53b90, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b6ee364cfeb342619227fbe795f1cff1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/45e4414a41d040eb9ee64a0291e7f39f] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=111.4 K 2023-04-07 09:56:18,205 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting ee0c8a8da7f84e8688033a240d1cfa55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680861376077 2023-04-07 09:56:18,206 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 33aa594f8aa94829a238fe160fd53b90, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680861375740 2023-04-07 09:56:18,206 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 80ecd05d750d409aba071bfb1f9362e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680861378059 2023-04-07 09:56:18,206 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b6ee364cfeb342619227fbe795f1cff1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680861376077 2023-04-07 09:56:18,208 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 45e4414a41d040eb9ee64a0291e7f39f, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680861378059 2023-04-07 09:56:18,224 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#B#compaction#662 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:18,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:18,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861378158, max timestamp=1680861378228Num uniques 1018; 2023-04-07 09:56:18,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:56:18,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861378158, max timestamp=1680861378229Num uniques 1018; 2023-04-07 09:56:18,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:56:18,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680861378158, max timestamp=1680861378229Num uniques 1018; 2023-04-07 09:56:18,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-07 09:56:18,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=64.85 KB heapSize=178.80 KB 2023-04-07 09:56:18,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:18,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:18,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:18,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:18,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:18,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:18,252 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:18,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:18,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:18,258 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#A#compaction#663 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:18,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:18,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:18,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861378257, max timestamp=1680861378267Num uniques 859; 2023-04-07 09:56:18,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:56:18,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861378257, max timestamp=1680861378267Num uniques 859; 2023-04-07 09:56:18,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680861378257, max timestamp=1680861378267Num uniques 859; 2023-04-07 09:56:18,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:56:18,269 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:18,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:18,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:18,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-07 09:56:18,269 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,270 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,271 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,271 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:18,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:18,277 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:18,277 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,277 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:18,278 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,278 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:18,278 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:18,278 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:18,278 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,281 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/25de17790e7948dd85bd6dfd6571f855 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/25de17790e7948dd85bd6dfd6571f855 2023-04-07 09:56:18,285 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/B of 448acd9b275d13bcb18a1b50f0aa55df into 25de17790e7948dd85bd6dfd6571f855(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:18,285 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:18,285 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/B, priority=13, startTime=1680861378201; duration=0sec 2023-04-07 09:56:18,285 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:18,285 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-07 09:56:18,286 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-07 09:56:18,286 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/C is initiating minor compaction (all files) 2023-04-07 09:56:18,287 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/C in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,287 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/6d282e95f61a45b1ae15e0742b273f17, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/09d9cd7b5daf42a781f1214102b93b28, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a1c96113a1ba45c382b0d5d612c63142] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=36.4 K 2023-04-07 09:56:18,287 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 6d282e95f61a45b1ae15e0742b273f17, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680861375740 2023-04-07 09:56:18,287 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 09d9cd7b5daf42a781f1214102b93b28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680861376077 2023-04-07 09:56:18,287 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting a1c96113a1ba45c382b0d5d612c63142, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680861378059 2023-04-07 09:56:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:18,294 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,294 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:18,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44b1e5cb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:18,295 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:18,295 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,296 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:18,296 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:18,297 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#C#compaction#665 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:18,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,298 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:18,300 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,300 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:18,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:18,302 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:18,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:18,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:18,302 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:18,302 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,304 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,305 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,306 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,309 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,309 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,309 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,309 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,309 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:18,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,309 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,310 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,309 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,310 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:18,310 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:18,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861438305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861438308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9975, totalHeapSize=22912, min timestamp=1680861378272, max timestamp=1680861378310Num uniques 1068; 2023-04-07 09:56:18,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48905, new segment data size=9975, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-07 09:56:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9977, totalHeapSize=22912, min timestamp=1680861378272, max timestamp=1680861378310Num uniques 1068; 2023-04-07 09:56:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48907, new segment data size=9977, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-07 09:56:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680861378272, max timestamp=1680861378310Num uniques 1068; 2023-04-07 09:56:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-07 09:56:18,329 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/0afb4b815b13464baeb2222d8fb2cbda as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/0afb4b815b13464baeb2222d8fb2cbda 2023-04-07 09:56:18,333 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/C of 448acd9b275d13bcb18a1b50f0aa55df into 0afb4b815b13464baeb2222d8fb2cbda(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:18,333 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:18,333 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/C, priority=13, startTime=1680861378202; duration=0sec 2023-04-07 09:56:18,334 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:18,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680861378334, max timestamp=1680861378349Num uniques 1077; 2023-04-07 09:56:18,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-07 09:56:18,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680861378334, max timestamp=1680861378349Num uniques 1077; 2023-04-07 09:56:18,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680861378334, max timestamp=1680861378349Num uniques 1077; 2023-04-07 09:56:18,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-07 09:56:18,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-07 09:56:18,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1680861378378, max timestamp=1680861378384Num uniques 1136; 2023-04-07 09:56:18,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861438382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52016, new segment data size=8506, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-07 09:56:18,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8515, totalHeapSize=19584, min timestamp=1680861378378, max timestamp=1680861378384Num uniques 1136; 2023-04-07 09:56:18,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52025, new segment data size=8515, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-07 09:56:18,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1680861378378, max timestamp=1680861378384Num uniques 1136; 2023-04-07 09:56:18,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52016, new segment data size=8506, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-07 09:56:18,410 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:18,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:18,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:18,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:18,414 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:18,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:18,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:18,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,414 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,415 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,415 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,416 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:18,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:18,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:18,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680861378405, max timestamp=1680861378416Num uniques 1045; 2023-04-07 09:56:18,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-07 09:56:18,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680861378405, max timestamp=1680861378416Num uniques 1045; 2023-04-07 09:56:18,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-07 09:56:18,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680861378405, max timestamp=1680861378416Num uniques 1045; 2023-04-07 09:56:18,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-07 09:56:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:18,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:18,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:18,424 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:18,424 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:18,426 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,426 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,427 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:18,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49b24b91[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:18,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:18,429 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,430 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:18,431 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,431 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:18,431 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,431 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:18,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:18,434 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:18,434 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:18,434 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:18,434 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:18,434 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,436 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,436 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,436 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,436 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,437 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,445 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:18,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,445 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,446 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:18,446 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:18,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,446 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38310 deadline: 1680861438464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861438464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38356 deadline: 1680861438464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861438468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861438489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,546 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:18,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:18,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:18,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:18,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:18,552 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:18,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:18,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:18,552 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,553 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,554 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,554 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:18,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:18,560 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,560 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:18,560 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:18,561 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:18,562 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,564 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:18,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6415f59a[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:18,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:18,566 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,567 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,567 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,567 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:18,567 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:18,570 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:18,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,570 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38310 deadline: 1680861438570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861438570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38356 deadline: 1680861438570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:18,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:18,573 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:18,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:18,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:18,574 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:18,574 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861438571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,576 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,578 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,579 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,581 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:18,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,582 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:18,582 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,584 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,682 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:18,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:18,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:18,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:18,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:18,685 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:18,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:18,685 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:18,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,687 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,687 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:18,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:18,688 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,688 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:18,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:18,689 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:18,689 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:18,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,691 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5e54bd[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:18,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:18,693 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,697 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861438691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:18,697 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:18,698 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,698 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:18,699 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,702 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040736e447325dfb4bc982f5b46ba2e1640f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040736e447325dfb4bc982f5b46ba2e1640f 2023-04-07 09:56:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:18,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:18,705 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:18,705 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:18,705 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:18,705 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:18,705 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,707 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,708 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,709 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,710 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/ff045cb8969f4acf819aba8b218c2445 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff045cb8969f4acf819aba8b218c2445 2023-04-07 09:56:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,713 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:18,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,714 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:18,713 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,714 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,714 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:18,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,714 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/A of 448acd9b275d13bcb18a1b50f0aa55df into ff045cb8969f4acf819aba8b218c2445(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:18,715 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:18,715 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/A, priority=13, startTime=1680861378201; duration=0sec 2023-04-07 09:56:18,715 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:18,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861438773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38356 deadline: 1680861438774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38310 deadline: 1680861438774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:18,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861438777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,814 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:18,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:18,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:18,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:18,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:18,818 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:18,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:18,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:18,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,818 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,819 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,820 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,821 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:18,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:18,822 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,822 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:18,822 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:18,822 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,823 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,824 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248a6b20[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:18,825 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:18,826 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,827 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:18,827 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,827 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,829 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:18,832 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:18,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:18,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:18,832 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:18,832 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,834 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,835 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,836 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,842 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:18,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,842 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:18,842 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:18,842 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,843 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,843 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,843 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,943 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:18,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:18,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:18,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:18,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:18,952 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:18,952 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,952 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:18,952 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:18,952 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,952 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,955 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:18,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:18,955 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,956 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:18,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:18,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:18,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:18,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:18,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:18,957 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:18,976 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,976 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:18,976 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:18,976 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,978 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,978 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,979 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:18,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3f52e3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:18,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:18,980 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,981 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:18,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,982 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:18,982 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:18,984 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:18,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:18,989 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:18,989 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:18,989 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:18,989 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:18,989 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:18,993 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,993 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:18,993 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:18,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:18,994 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,997 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:18,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:18,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:18,999 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:18,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:19,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:18,999 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,000 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,000 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,000 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,000 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:19,000 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:19,000 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:19,000 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:19,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:19,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:19,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861438999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861439076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38356 deadline: 1680861439078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38310 deadline: 1680861439078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861439079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,100 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:19,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:19,104 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:19,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:19,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:19,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:19,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,104 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,106 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:19,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:19,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:19,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:19,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:19,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:19,107 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:19,108 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:19,108 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:19,108 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:19,108 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:19,108 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:19,108 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=507, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/278bbf1e8bf7485b8ea669b155cc5ac4 2023-04-07 09:56:19,110 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:19,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:19,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34cdd1bb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:19,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:19,113 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:19,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,116 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:19,124 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:19,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:19,127 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:19,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:19,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:19,127 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:19,127 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:19,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,129 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/3b70bd76b39247479136ff045b820524 2023-04-07 09:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:19,130 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:19,132 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:19,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:19,136 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,136 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:19,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:19,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:19,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:19,137 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:19,137 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:19,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:19,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/743be0e96ec84edc9debaa03622260c0 2023-04-07 09:56:19,144 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/278bbf1e8bf7485b8ea669b155cc5ac4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/278bbf1e8bf7485b8ea669b155cc5ac4 2023-04-07 09:56:19,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/278bbf1e8bf7485b8ea669b155cc5ac4, entries=150, sequenceid=507, filesize=25.4 K 2023-04-07 09:56:19,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/3b70bd76b39247479136ff045b820524 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/3b70bd76b39247479136ff045b820524 2023-04-07 09:56:19,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/3b70bd76b39247479136ff045b820524, entries=150, sequenceid=507, filesize=11.9 K 2023-04-07 09:56:19,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/743be0e96ec84edc9debaa03622260c0 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/743be0e96ec84edc9debaa03622260c0 2023-04-07 09:56:19,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/743be0e96ec84edc9debaa03622260c0, entries=150, sequenceid=507, filesize=11.9 K 2023-04-07 09:56:19,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=154.31 KB/158010 for 448acd9b275d13bcb18a1b50f0aa55df in 927ms, sequenceid=507, compaction requested=false 2023-04-07 09:56:19,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:19,237 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:19,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:19,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:19,240 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:19,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:19,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:19,240 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:19,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,241 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,243 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:19,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:19,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,243 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:19,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:19,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:19,244 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:19,244 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:19,245 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=154.31 KB heapSize=427.31 KB 2023-04-07 09:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:19,245 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:19,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:19,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:19,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861439520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861439578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38310 deadline: 1680861439582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38356 deadline: 1680861439584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861439586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861439623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:19,657 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e449d79b94a64f12b87afbdcbc14df8a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e449d79b94a64f12b87afbdcbc14df8a 2023-04-07 09:56:19,661 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=615, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/30c19cfc9fc4472798dc38575fc1b07c 2023-04-07 09:56:19,669 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/79b41254e171416c8b1147408aefed99 2023-04-07 09:56:19,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:19,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861439827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,078 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/8b8559cec5294d7ea354d28a4052491d 2023-04-07 09:56:20,082 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/30c19cfc9fc4472798dc38575fc1b07c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/30c19cfc9fc4472798dc38575fc1b07c 2023-04-07 09:56:20,085 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/30c19cfc9fc4472798dc38575fc1b07c, entries=150, sequenceid=615, filesize=25.4 K 2023-04-07 09:56:20,086 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/79b41254e171416c8b1147408aefed99 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/79b41254e171416c8b1147408aefed99 2023-04-07 09:56:20,092 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/79b41254e171416c8b1147408aefed99, entries=150, sequenceid=615, filesize=11.9 K 2023-04-07 09:56:20,093 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/8b8559cec5294d7ea354d28a4052491d as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8b8559cec5294d7ea354d28a4052491d 2023-04-07 09:56:20,096 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8b8559cec5294d7ea354d28a4052491d, entries=150, sequenceid=615, filesize=11.9 K 2023-04-07 09:56:20,097 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~427.27 KB/437520, currentSize=33.54 KB/34350 for 448acd9b275d13bcb18a1b50f0aa55df in 852ms, sequenceid=615, compaction requested=true 2023-04-07 09:56:20,097 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:20,097 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:20,097 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:20,099 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,099 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292b8023[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:20,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:20,101 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,102 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:20,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:20,102 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:20,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,103 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:20,104 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,104 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:20,104 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,104 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:20,106 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:20,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:20,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:20,106 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:20,107 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:20,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,109 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,110 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,111 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:20,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:20,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,115 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,115 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:20,115 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,115 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:20,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:20,143 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-07 09:56:20,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:20,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:20,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:20,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:20,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:20,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:20,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861380152, max timestamp=1680861380201Num uniques 859; 2023-04-07 09:56:20,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:20,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861380152, max timestamp=1680861380201Num uniques 859; 2023-04-07 09:56:20,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:20,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680861380152, max timestamp=1680861380201Num uniques 859; 2023-04-07 09:56:20,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-07 09:56:20,215 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:20,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:20,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:20,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:20,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:20,219 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:20,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:20,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:20,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,219 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,220 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,220 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:20,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:20,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:20,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:20,222 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:20,222 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:20,227 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,227 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0f2d4d[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:20,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:20,229 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,230 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:20,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,230 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:20,231 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,231 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:20,233 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:20,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:20,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:20,233 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:20,233 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:20,234 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,235 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,235 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,235 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,236 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,239 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:20,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:20,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,239 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:20,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:20,240 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:20,239 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,240 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:20,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861440268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,340 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:20,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:20,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:20,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:20,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:20,343 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:20,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:20,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:20,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,343 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,344 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,345 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,345 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:20,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:20,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:20,355 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,355 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:20,355 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:20,355 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:20,355 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:20,357 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,357 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba2c425[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:20,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:20,359 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,361 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:20,361 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,362 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:20,363 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:20,366 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:20,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:20,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:20,366 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:20,366 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:20,368 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,368 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,368 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,368 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,369 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,371 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:20,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:20,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:20,371 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,372 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,372 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,372 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,372 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:20,372 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:20,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:20,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861440374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,473 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:20,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:20,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:20,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:20,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:20,479 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:20,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:20,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:20,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,479 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,481 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,481 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,482 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:20,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:20,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:20,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:20,485 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:20,486 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:20,488 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,488 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13886e3[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:20,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:20,490 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,492 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,492 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:20,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,494 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:20,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:20,496 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:20,496 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:20,496 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:20,497 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:20,497 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:20,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,498 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,499 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,500 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,504 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:20,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,504 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:20,505 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:20,504 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,505 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,573 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407e81483028e4c4b32b838d0b21b2b97ab to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e81483028e4c4b32b838d0b21b2b97ab 2023-04-07 09:56:20,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:20,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35881] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38350 deadline: 1680861440580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,586 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=625, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/b42c931fb01c4388b821d3aabbd56195 2023-04-07 09:56:20,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:20,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38356 deadline: 1680861440587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38310 deadline: 1680861440589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38308 deadline: 1680861440589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-07 09:56:20,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35881] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38334 deadline: 1680861440590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,605 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:20,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:20,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:20,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:20,615 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,615 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,617 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,617 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,618 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:20,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-07 09:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:20,624 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,625 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:20,632 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,632 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:20,632 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:20,632 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/b5232804df9f42fe9a5c0580975ad99f 2023-04-07 09:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:20,635 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,635 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,636 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,636 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e7247f[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:20,638 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:20,638 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,643 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:20,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:20,643 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:20,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,644 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:20,645 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:20,645 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,653 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/67d45662e46f4e8899d079c64bbb93a3 2023-04-07 09:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:20,676 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:20,676 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:20,676 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:20,676 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:20,676 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:20,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/b42c931fb01c4388b821d3aabbd56195 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b42c931fb01c4388b821d3aabbd56195 2023-04-07 09:56:20,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,683 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:20,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,684 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,686 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:20,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b42c931fb01c4388b821d3aabbd56195, entries=150, sequenceid=625, filesize=25.4 K 2023-04-07 09:56:20,690 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,690 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:20,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/b5232804df9f42fe9a5c0580975ad99f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/b5232804df9f42fe9a5c0580975ad99f 2023-04-07 09:56:20,690 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:20,690 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:20,691 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:20,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:20,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:20,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:20,691 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:20,691 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:20,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/b5232804df9f42fe9a5c0580975ad99f, entries=150, sequenceid=625, filesize=11.9 K 2023-04-07 09:56:20,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/67d45662e46f4e8899d079c64bbb93a3 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/67d45662e46f4e8899d079c64bbb93a3 2023-04-07 09:56:20,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/67d45662e46f4e8899d079c64bbb93a3, entries=150, sequenceid=625, filesize=11.9 K 2023-04-07 09:56:20,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 448acd9b275d13bcb18a1b50f0aa55df in 556ms, sequenceid=625, compaction requested=true 2023-04-07 09:56:20,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:20,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:20,699 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:56:20,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:20,699 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:56:20,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:20,700 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:56:20,700 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/A is initiating minor compaction (all files) 2023-04-07 09:56:20,701 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/A in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,701 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff045cb8969f4acf819aba8b218c2445, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/278bbf1e8bf7485b8ea669b155cc5ac4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/30c19cfc9fc4472798dc38575fc1b07c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b42c931fb01c4388b821d3aabbd56195] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=102.6 K 2023-04-07 09:56:20,701 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:56:20,701 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/B is initiating minor compaction (all files) 2023-04-07 09:56:20,701 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting ff045cb8969f4acf819aba8b218c2445, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680861378059 2023-04-07 09:56:20,701 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/B in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,701 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/25de17790e7948dd85bd6dfd6571f855, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/3b70bd76b39247479136ff045b820524, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/79b41254e171416c8b1147408aefed99, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/b5232804df9f42fe9a5c0580975ad99f] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=48.4 K 2023-04-07 09:56:20,702 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 278bbf1e8bf7485b8ea669b155cc5ac4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680861378174 2023-04-07 09:56:20,702 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 25de17790e7948dd85bd6dfd6571f855, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680861378059 2023-04-07 09:56:20,702 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting 30c19cfc9fc4472798dc38575fc1b07c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680861378433 2023-04-07 09:56:20,702 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 3b70bd76b39247479136ff045b820524, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680861378174 2023-04-07 09:56:20,703 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 79b41254e171416c8b1147408aefed99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680861378433 2023-04-07 09:56:20,703 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] compactions.Compactor(207): Compacting b42c931fb01c4388b821d3aabbd56195, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861379519 2023-04-07 09:56:20,703 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting b5232804df9f42fe9a5c0580975ad99f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861379519 2023-04-07 09:56:20,723 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#B#compaction#675 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:20,730 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#A#compaction#674 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:20,771 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/1f75f9b4d0614625b8f62a17658888f4 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1f75f9b4d0614625b8f62a17658888f4 2023-04-07 09:56:20,777 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/B of 448acd9b275d13bcb18a1b50f0aa55df into 1f75f9b4d0614625b8f62a17658888f4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:20,777 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:20,777 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/B, priority=12, startTime=1680861380699; duration=0sec 2023-04-07 09:56:20,777 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-07 09:56:20,777 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-07 09:56:20,778 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-07 09:56:20,779 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1912): 448acd9b275d13bcb18a1b50f0aa55df/C is initiating minor compaction (all files) 2023-04-07 09:56:20,779 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 448acd9b275d13bcb18a1b50f0aa55df/C in TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,779 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/0afb4b815b13464baeb2222d8fb2cbda, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/743be0e96ec84edc9debaa03622260c0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8b8559cec5294d7ea354d28a4052491d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/67d45662e46f4e8899d079c64bbb93a3] into tmpdir=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp, totalSize=48.4 K 2023-04-07 09:56:20,779 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 0afb4b815b13464baeb2222d8fb2cbda, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1680861378059 2023-04-07 09:56:20,779 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 743be0e96ec84edc9debaa03622260c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680861378174 2023-04-07 09:56:20,780 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 8b8559cec5294d7ea354d28a4052491d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680861378433 2023-04-07 09:56:20,780 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] compactions.Compactor(207): Compacting 67d45662e46f4e8899d079c64bbb93a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680861379519 2023-04-07 09:56:20,789 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] throttle.PressureAwareThroughputController(145): 448acd9b275d13bcb18a1b50f0aa55df#C#compaction#676 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-07 09:56:20,791 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:20,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:20,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-07 09:56:20,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-07 09:56:20,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-07 09:56:20,798 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-07 09:56:20,798 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,798 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-07 09:56:20,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-07 09:56:20,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,799 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,802 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:20,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:20,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:20,802 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-07 09:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:20,804 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/1814ac1356ba4983bb11bb9c6baa8890 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1814ac1356ba4983bb11bb9c6baa8890 2023-04-07 09:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-07 09:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:20,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-07 09:56:20,804 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/555e2241c90346918efa867907e84bdb as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/555e2241c90346918efa867907e84bdb 2023-04-07 09:56:20,810 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/A of 448acd9b275d13bcb18a1b50f0aa55df into 555e2241c90346918efa867907e84bdb(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:20,810 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:20,811 INFO [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/A, priority=12, startTime=1680861380699; duration=0sec 2023-04-07 09:56:20,811 DEBUG [RS:0;jenkins-hbase4:35881-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:20,811 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 448acd9b275d13bcb18a1b50f0aa55df/C of 448acd9b275d13bcb18a1b50f0aa55df into 1814ac1356ba4983bb11bb9c6baa8890(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-07 09:56:20,811 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:20,811 INFO [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df., storeName=448acd9b275d13bcb18a1b50f0aa55df/C, priority=12, startTime=1680861380699; duration=0sec 2023-04-07 09:56:20,811 DEBUG [RS:0;jenkins-hbase4:35881-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-07 09:56:20,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-07 09:56:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-07 09:56:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-07 09:56:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-07 09:56:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-07 09:56:20,812 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-07 09:56:20,812 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:20,813 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. started... 2023-04-07 09:56:20,813 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-07 09:56:20,813 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:20,813 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:20,813 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:20,813 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:20,813 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:20,813 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-07 09:56:20,825 DEBUG [Thread-4403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:49660 2023-04-07 09:56:20,825 DEBUG [Thread-4403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:20,826 DEBUG [Thread-4407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:49660 2023-04-07 09:56:20,826 DEBUG [Thread-4407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:20,827 DEBUG [Thread-4405] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:49660 2023-04-07 09:56:20,827 DEBUG [Thread-4405] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:20,827 DEBUG [Thread-4409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:49660 2023-04-07 09:56:20,827 DEBUG [Thread-4409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:20,829 DEBUG [Thread-4411] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eedd898 to 127.0.0.1:49660 2023-04-07 09:56:20,829 DEBUG [Thread-4411] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:20,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35881] regionserver.HRegion(9158): Flush requested on 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:20,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. as already flushing 2023-04-07 09:56:20,885 DEBUG [Thread-4400] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:49660 2023-04-07 09:56:20,885 DEBUG [Thread-4400] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:21,224 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407d98fc09dceae4181981111fda6e0d1ed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d98fc09dceae4181981111fda6e0d1ed 2023-04-07 09:56:21,628 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=665, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/184af33cb6784de7abc8c2958f8dc241 2023-04-07 09:56:22,036 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/d3a4e70b58c44acebaff8b663c29f08f 2023-04-07 09:56:22,444 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/f18f75175821483292ca8eca6f09001a 2023-04-07 09:56:22,447 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/184af33cb6784de7abc8c2958f8dc241 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/184af33cb6784de7abc8c2958f8dc241 2023-04-07 09:56:22,449 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/184af33cb6784de7abc8c2958f8dc241, entries=150, sequenceid=665, filesize=25.4 K 2023-04-07 09:56:22,450 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/d3a4e70b58c44acebaff8b663c29f08f as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d3a4e70b58c44acebaff8b663c29f08f 2023-04-07 09:56:22,453 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d3a4e70b58c44acebaff8b663c29f08f, entries=150, sequenceid=665, filesize=11.9 K 2023-04-07 09:56:22,454 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/f18f75175821483292ca8eca6f09001a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f18f75175821483292ca8eca6f09001a 2023-04-07 09:56:22,456 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f18f75175821483292ca8eca6f09001a, entries=150, sequenceid=665, filesize=11.9 K 2023-04-07 09:56:22,457 INFO [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=6.71 KB/6870 for 448acd9b275d13bcb18a1b50f0aa55df in 1644ms, sequenceid=665, compaction requested=false 2023-04-07 09:56:22,457 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:22,457 DEBUG [rs(jenkins-hbase4.apache.org,35881,1680861218288)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-07 09:56:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-07 09:56:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-07 09:56:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-07 09:56:22,458 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:22,458 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-07 09:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:22,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35881,1680861218288' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-07 09:56:22,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a50aeb[Count = 0] remaining members to acquire global barrier 2023-04-07 09:56:22,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-07 09:56:22,460 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,461 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-07 09:56:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-07 09:56:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35881,1680861218288' in zk 2023-04-07 09:56:22,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,461 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-07 09:56:22,462 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-07 09:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,35881,1680861218288' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-07 09:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35881,1680861218288': 2023-04-07 09:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35881,1680861218288' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-07 09:56:22,464 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-07 09:56:22,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-07 09:56:22,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-07 09:56:22,464 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-07 09:56:22,464 INFO [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-07 09:56:22,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:22,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:22,467 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-07 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-07 09:56:22,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:22,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-07 09:56:22,468 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-07 09:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:22,469 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-07 09:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-07 09:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-07 09:56:22,470 DEBUG [(jenkins-hbase4.apache.org,46673,1680861217027)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-07 09:56:22,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-07 09:56:22,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-07 09:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-07 09:56:22,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-07 09:56:22,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-07 09:56:22,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-07 09:56:22,470 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:22,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-07 09:56:22,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-07 09:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-07 09:56:22,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,471 DEBUG [Thread-4402] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-07 09:56:22,471 DEBUG [Thread-4402] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-07 09:56:22,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-07 09:56:22,471 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-07 09:56:22,571 DEBUG [Thread-4402] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-07 09:56:22,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-07 09:56:22,596 DEBUG [Thread-4394] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:49660 2023-04-07 09:56:22,596 DEBUG [Thread-4394] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:22,603 DEBUG [Thread-4392] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:49660 2023-04-07 09:56:22,604 DEBUG [Thread-4392] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:22,611 DEBUG [Thread-4398] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:49660 2023-04-07 09:56:22,611 DEBUG [Thread-4398] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:22,615 DEBUG [Thread-4396] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:49660 2023-04-07 09:56:22,615 DEBUG [Thread-4396] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 4033 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 3963 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 3912 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 3998 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(398): read 3936 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-07 09:56:22,615 INFO [Listener at localhost/39511] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 09:56:22,616 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:49660 2023-04-07 09:56:22,616 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:22,616 INFO [Listener at localhost/39511] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-07 09:56:22,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-07 09:56:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:22,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 09:56:22,619 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861382619"}]},"ts":"1680861382619"} 2023-04-07 09:56:22,620 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-07 09:56:22,621 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-07 09:56:22,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, UNASSIGN}] 2023-04-07 09:56:22,623 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, UNASSIGN 2023-04-07 09:56:22,623 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:22,623 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861382623"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680861382623"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680861382623"}]},"ts":"1680861382623"} 2023-04-07 09:56:22,624 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288}] 2023-04-07 09:56:22,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 09:56:22,775 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:22,775 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 448acd9b275d13bcb18a1b50f0aa55df, disabling compactions & flushes 2023-04-07 09:56:22,775 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. after waiting 0 ms 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:22,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 448acd9b275d13bcb18a1b50f0aa55df 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=A 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=B 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 448acd9b275d13bcb18a1b50f0aa55df, store=C 2023-04-07 09:56:22,776 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-07 09:56:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 09:56:23,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230407da454546d5634daeaeef70f0b40fb8bc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407da454546d5634daeaeef70f0b40fb8bc 2023-04-07 09:56:23,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 09:56:23,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=673, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/704afeb88bba49e0a7c5e98b237f8ac1 2023-04-07 09:56:23,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 09:56:24,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/ffab3e8c87b141c8bb15666a9b9ea45e 2023-04-07 09:56:24,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/75b6345a090045499092e2731b5c966a 2023-04-07 09:56:24,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/A/704afeb88bba49e0a7c5e98b237f8ac1 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/704afeb88bba49e0a7c5e98b237f8ac1 2023-04-07 09:56:24,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/704afeb88bba49e0a7c5e98b237f8ac1, entries=100, sequenceid=673, filesize=18.6 K 2023-04-07 09:56:24,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/B/ffab3e8c87b141c8bb15666a9b9ea45e as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffab3e8c87b141c8bb15666a9b9ea45e 2023-04-07 09:56:24,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffab3e8c87b141c8bb15666a9b9ea45e, entries=100, sequenceid=673, filesize=9.5 K 2023-04-07 09:56:24,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/.tmp/C/75b6345a090045499092e2731b5c966a as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/75b6345a090045499092e2731b5c966a 2023-04-07 09:56:24,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/75b6345a090045499092e2731b5c966a, entries=100, sequenceid=673, filesize=9.5 K 2023-04-07 09:56:24,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 448acd9b275d13bcb18a1b50f0aa55df in 1657ms, sequenceid=673, compaction requested=true 2023-04-07 09:56:24,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6479e0e3a862439b84eecfad30e501dd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6b771ef0930f4cafbc86d7373cde8838, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/83aed4a4f4644315870bffae9708da8b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/3fe134e780cf4f66945541f4807ca5a1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/29f6e2f04fd545ba95094f690915399c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/9fad943c543147baad71938b36692684, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8fcfb19c71c7437e9ac7b22f1b6ec95e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/bd0abd8935e34543a7834ab5e05afa05, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/14a7cc3242174f04a71d58a7a6da3d9c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/40f03b4909bf47cda272b91b3ee31023, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/abe9e3a808344d27889febc59f673d81, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5e3e58b934264c8d9634a1d65a87f918, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/13df56c11981450f8908dac1b2ed8387, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/a0182f3ec71543f1a472c4fe56169273, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/7fa7783763b34c4c9fd5f565650b24be, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/fb691cfe282b472c9a8eadb85196c908, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff9b17c8e77444b2a3df4434404b5129, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8c4a7f64b5304b6997e9578015e91756, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/0c5773cbae3c4264832133894317e776, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5dd2c4f6ec984570988e580618197d0d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b4d1221159d8451c998bc8776e20de69, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/90db6084db2847cc9752c01207e1622e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/33aa594f8aa94829a238fe160fd53b90, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b6ee364cfeb342619227fbe795f1cff1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/45e4414a41d040eb9ee64a0291e7f39f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff045cb8969f4acf819aba8b218c2445, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/278bbf1e8bf7485b8ea669b155cc5ac4, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/30c19cfc9fc4472798dc38575fc1b07c, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b42c931fb01c4388b821d3aabbd56195] to archive 2023-04-07 09:56:24,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:56:24,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6479e0e3a862439b84eecfad30e501dd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6479e0e3a862439b84eecfad30e501dd 2023-04-07 09:56:24,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6b771ef0930f4cafbc86d7373cde8838 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/6b771ef0930f4cafbc86d7373cde8838 2023-04-07 09:56:24,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/83aed4a4f4644315870bffae9708da8b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/83aed4a4f4644315870bffae9708da8b 2023-04-07 09:56:24,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/3fe134e780cf4f66945541f4807ca5a1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/3fe134e780cf4f66945541f4807ca5a1 2023-04-07 09:56:24,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/29f6e2f04fd545ba95094f690915399c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/29f6e2f04fd545ba95094f690915399c 2023-04-07 09:56:24,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/9fad943c543147baad71938b36692684 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/9fad943c543147baad71938b36692684 2023-04-07 09:56:24,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8fcfb19c71c7437e9ac7b22f1b6ec95e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8fcfb19c71c7437e9ac7b22f1b6ec95e 2023-04-07 09:56:24,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/bd0abd8935e34543a7834ab5e05afa05 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/bd0abd8935e34543a7834ab5e05afa05 2023-04-07 09:56:24,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/14a7cc3242174f04a71d58a7a6da3d9c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/14a7cc3242174f04a71d58a7a6da3d9c 2023-04-07 09:56:24,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/40f03b4909bf47cda272b91b3ee31023 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/40f03b4909bf47cda272b91b3ee31023 2023-04-07 09:56:24,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/abe9e3a808344d27889febc59f673d81 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/abe9e3a808344d27889febc59f673d81 2023-04-07 09:56:24,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5e3e58b934264c8d9634a1d65a87f918 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5e3e58b934264c8d9634a1d65a87f918 2023-04-07 09:56:24,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/13df56c11981450f8908dac1b2ed8387 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/13df56c11981450f8908dac1b2ed8387 2023-04-07 09:56:24,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/a0182f3ec71543f1a472c4fe56169273 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/a0182f3ec71543f1a472c4fe56169273 2023-04-07 09:56:24,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/7fa7783763b34c4c9fd5f565650b24be to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/7fa7783763b34c4c9fd5f565650b24be 2023-04-07 09:56:24,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/fb691cfe282b472c9a8eadb85196c908 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/fb691cfe282b472c9a8eadb85196c908 2023-04-07 09:56:24,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff9b17c8e77444b2a3df4434404b5129 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff9b17c8e77444b2a3df4434404b5129 2023-04-07 09:56:24,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8c4a7f64b5304b6997e9578015e91756 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/8c4a7f64b5304b6997e9578015e91756 2023-04-07 09:56:24,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/0c5773cbae3c4264832133894317e776 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/0c5773cbae3c4264832133894317e776 2023-04-07 09:56:24,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5dd2c4f6ec984570988e580618197d0d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/5dd2c4f6ec984570988e580618197d0d 2023-04-07 09:56:24,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b4d1221159d8451c998bc8776e20de69 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b4d1221159d8451c998bc8776e20de69 2023-04-07 09:56:24,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/90db6084db2847cc9752c01207e1622e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/90db6084db2847cc9752c01207e1622e 2023-04-07 09:56:24,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/33aa594f8aa94829a238fe160fd53b90 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/33aa594f8aa94829a238fe160fd53b90 2023-04-07 09:56:24,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b6ee364cfeb342619227fbe795f1cff1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b6ee364cfeb342619227fbe795f1cff1 2023-04-07 09:56:24,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/45e4414a41d040eb9ee64a0291e7f39f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/45e4414a41d040eb9ee64a0291e7f39f 2023-04-07 09:56:24,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff045cb8969f4acf819aba8b218c2445 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/ff045cb8969f4acf819aba8b218c2445 2023-04-07 09:56:24,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/278bbf1e8bf7485b8ea669b155cc5ac4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/278bbf1e8bf7485b8ea669b155cc5ac4 2023-04-07 09:56:24,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/30c19cfc9fc4472798dc38575fc1b07c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/30c19cfc9fc4472798dc38575fc1b07c 2023-04-07 09:56:24,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b42c931fb01c4388b821d3aabbd56195 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/b42c931fb01c4388b821d3aabbd56195 2023-04-07 09:56:24,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/affdfdba46374a24b22840352c5a02e6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/860a6f060c524799b51f1c1aa80967bd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/83a0df2f5f34480687a78852d42a8efd, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffe6d7ed702e4903893904c1bd96cfd7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0115b1ac44014bd98e3732c1c1ff8136, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4ad0e307a18645ce9a7b621158a80fbf, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/7b980e29ee9d46788077dd19547cbdb6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8a4b8e66de5a4540867c4cf933e652c3, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/5d72254dea4844f1bb50bc8859dfafe1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8e42dc7e180d4c529853a24f8a4fec83, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/03eef0fec755435db9cb23053b44c6bb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/a957bb862fd4447d8f55562eb005a0c0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9dff6a69043d4aeeb8629185baff2fff, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/31e2df401b074f51bacefc80f2932189, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4e162f920ff34a38b4a08782f9c7b59f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0da994fe2b8e444ab910c9f66fc34e9f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d82506d46a9d4b24bccfd58ba146b077, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9a1b7f6e81c449afbfb29ecc0c5bb35d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/26e12a550d1147e7acf8d9d1cb0101ff, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1127d55a3ba54150871f8e134a7ac778, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/365f8e6577db423698f2fb9409d7c3f9, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/c13442f022a24fa5bb51d0dbd6d956a1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8735ef5cb1804340bb625e703a8ea348, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ee0c8a8da7f84e8688033a240d1cfa55, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/25de17790e7948dd85bd6dfd6571f855, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/80ecd05d750d409aba071bfb1f9362e7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/3b70bd76b39247479136ff045b820524, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/79b41254e171416c8b1147408aefed99, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/b5232804df9f42fe9a5c0580975ad99f] to archive 2023-04-07 09:56:24,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:56:24,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/affdfdba46374a24b22840352c5a02e6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/affdfdba46374a24b22840352c5a02e6 2023-04-07 09:56:24,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/860a6f060c524799b51f1c1aa80967bd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/860a6f060c524799b51f1c1aa80967bd 2023-04-07 09:56:24,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/83a0df2f5f34480687a78852d42a8efd to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/83a0df2f5f34480687a78852d42a8efd 2023-04-07 09:56:24,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffe6d7ed702e4903893904c1bd96cfd7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffe6d7ed702e4903893904c1bd96cfd7 2023-04-07 09:56:24,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0115b1ac44014bd98e3732c1c1ff8136 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0115b1ac44014bd98e3732c1c1ff8136 2023-04-07 09:56:24,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4ad0e307a18645ce9a7b621158a80fbf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4ad0e307a18645ce9a7b621158a80fbf 2023-04-07 09:56:24,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/7b980e29ee9d46788077dd19547cbdb6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/7b980e29ee9d46788077dd19547cbdb6 2023-04-07 09:56:24,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8a4b8e66de5a4540867c4cf933e652c3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8a4b8e66de5a4540867c4cf933e652c3 2023-04-07 09:56:24,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/5d72254dea4844f1bb50bc8859dfafe1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/5d72254dea4844f1bb50bc8859dfafe1 2023-04-07 09:56:24,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8e42dc7e180d4c529853a24f8a4fec83 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8e42dc7e180d4c529853a24f8a4fec83 2023-04-07 09:56:24,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/03eef0fec755435db9cb23053b44c6bb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/03eef0fec755435db9cb23053b44c6bb 2023-04-07 09:56:24,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/a957bb862fd4447d8f55562eb005a0c0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/a957bb862fd4447d8f55562eb005a0c0 2023-04-07 09:56:24,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9dff6a69043d4aeeb8629185baff2fff to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9dff6a69043d4aeeb8629185baff2fff 2023-04-07 09:56:24,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/31e2df401b074f51bacefc80f2932189 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/31e2df401b074f51bacefc80f2932189 2023-04-07 09:56:24,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4e162f920ff34a38b4a08782f9c7b59f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/4e162f920ff34a38b4a08782f9c7b59f 2023-04-07 09:56:24,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0da994fe2b8e444ab910c9f66fc34e9f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/0da994fe2b8e444ab910c9f66fc34e9f 2023-04-07 09:56:24,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d82506d46a9d4b24bccfd58ba146b077 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d82506d46a9d4b24bccfd58ba146b077 2023-04-07 09:56:24,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9a1b7f6e81c449afbfb29ecc0c5bb35d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/9a1b7f6e81c449afbfb29ecc0c5bb35d 2023-04-07 09:56:24,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/26e12a550d1147e7acf8d9d1cb0101ff to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/26e12a550d1147e7acf8d9d1cb0101ff 2023-04-07 09:56:24,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1127d55a3ba54150871f8e134a7ac778 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1127d55a3ba54150871f8e134a7ac778 2023-04-07 09:56:24,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/365f8e6577db423698f2fb9409d7c3f9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/365f8e6577db423698f2fb9409d7c3f9 2023-04-07 09:56:24,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/c13442f022a24fa5bb51d0dbd6d956a1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/c13442f022a24fa5bb51d0dbd6d956a1 2023-04-07 09:56:24,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8735ef5cb1804340bb625e703a8ea348 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/8735ef5cb1804340bb625e703a8ea348 2023-04-07 09:56:24,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ee0c8a8da7f84e8688033a240d1cfa55 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ee0c8a8da7f84e8688033a240d1cfa55 2023-04-07 09:56:24,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/25de17790e7948dd85bd6dfd6571f855 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/25de17790e7948dd85bd6dfd6571f855 2023-04-07 09:56:24,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/80ecd05d750d409aba071bfb1f9362e7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/80ecd05d750d409aba071bfb1f9362e7 2023-04-07 09:56:24,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/3b70bd76b39247479136ff045b820524 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/3b70bd76b39247479136ff045b820524 2023-04-07 09:56:24,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/79b41254e171416c8b1147408aefed99 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/79b41254e171416c8b1147408aefed99 2023-04-07 09:56:24,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/b5232804df9f42fe9a5c0580975ad99f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/b5232804df9f42fe9a5c0580975ad99f 2023-04-07 09:56:24,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/9777a0b2a40842af99939ef1549808c8, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8213919df9d6459bb79546180e34fd7f, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/d0b8f76e28df43bb92ef13e80510e7d1, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b83c9f6378834e8290b9709d9fc5c947, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f4caabc5a807423bb9cf09da455e4505, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b5ff43efec764631b8334a3dfe1416c7, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4472b330a1ff48c3937db8210c0c6883, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/087ce9702ac94c7bb2ab9c58c4726503, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/cb6c464667f44f33ab9a7e6404f1b07e, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/fb1dbb49de3744999342d29edf108542, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/68359d9e4a9f44e28c4b0c766619a3c6, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4146d4f46c6041a4b9093d91e6fff989, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/29aff34f714747b5b867607521ce6bac, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8cc4ec66894644c8bbea2085b2523444, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/478596edd77a4079b49f5e6f0e3b3892, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1687d48eea7847c7bfbd52174623f67b, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b773abbc8a11431d953ef669da263d3a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a5126f5ac4ac4e78846238dfe4d76661, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/02b46161d7f14ce0919ef581a27682fb, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/00fa8c9d68e84e7a81f5e9a9c7e6551a, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/6d282e95f61a45b1ae15e0742b273f17, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/e49e1628d4a941e3afa98034955ca944, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/09d9cd7b5daf42a781f1214102b93b28, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/0afb4b815b13464baeb2222d8fb2cbda, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a1c96113a1ba45c382b0d5d612c63142, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/743be0e96ec84edc9debaa03622260c0, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8b8559cec5294d7ea354d28a4052491d, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/67d45662e46f4e8899d079c64bbb93a3] to archive 2023-04-07 09:56:24,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-07 09:56:24,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4f1fdaf5a5ae441fa6cab6db8c0a3dbc 2023-04-07 09:56:24,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/9777a0b2a40842af99939ef1549808c8 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/9777a0b2a40842af99939ef1549808c8 2023-04-07 09:56:24,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8213919df9d6459bb79546180e34fd7f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8213919df9d6459bb79546180e34fd7f 2023-04-07 09:56:24,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/d0b8f76e28df43bb92ef13e80510e7d1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/d0b8f76e28df43bb92ef13e80510e7d1 2023-04-07 09:56:24,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b83c9f6378834e8290b9709d9fc5c947 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b83c9f6378834e8290b9709d9fc5c947 2023-04-07 09:56:24,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f4caabc5a807423bb9cf09da455e4505 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f4caabc5a807423bb9cf09da455e4505 2023-04-07 09:56:24,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b5ff43efec764631b8334a3dfe1416c7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b5ff43efec764631b8334a3dfe1416c7 2023-04-07 09:56:24,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4472b330a1ff48c3937db8210c0c6883 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4472b330a1ff48c3937db8210c0c6883 2023-04-07 09:56:24,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/087ce9702ac94c7bb2ab9c58c4726503 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/087ce9702ac94c7bb2ab9c58c4726503 2023-04-07 09:56:24,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/cb6c464667f44f33ab9a7e6404f1b07e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/cb6c464667f44f33ab9a7e6404f1b07e 2023-04-07 09:56:24,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/fb1dbb49de3744999342d29edf108542 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/fb1dbb49de3744999342d29edf108542 2023-04-07 09:56:24,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/68359d9e4a9f44e28c4b0c766619a3c6 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/68359d9e4a9f44e28c4b0c766619a3c6 2023-04-07 09:56:24,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4146d4f46c6041a4b9093d91e6fff989 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/4146d4f46c6041a4b9093d91e6fff989 2023-04-07 09:56:24,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/29aff34f714747b5b867607521ce6bac to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/29aff34f714747b5b867607521ce6bac 2023-04-07 09:56:24,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8cc4ec66894644c8bbea2085b2523444 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8cc4ec66894644c8bbea2085b2523444 2023-04-07 09:56:24,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/478596edd77a4079b49f5e6f0e3b3892 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/478596edd77a4079b49f5e6f0e3b3892 2023-04-07 09:56:24,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1687d48eea7847c7bfbd52174623f67b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1687d48eea7847c7bfbd52174623f67b 2023-04-07 09:56:24,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b773abbc8a11431d953ef669da263d3a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/b773abbc8a11431d953ef669da263d3a 2023-04-07 09:56:24,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a5126f5ac4ac4e78846238dfe4d76661 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a5126f5ac4ac4e78846238dfe4d76661 2023-04-07 09:56:24,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/02b46161d7f14ce0919ef581a27682fb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/02b46161d7f14ce0919ef581a27682fb 2023-04-07 09:56:24,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/00fa8c9d68e84e7a81f5e9a9c7e6551a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/00fa8c9d68e84e7a81f5e9a9c7e6551a 2023-04-07 09:56:24,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/6d282e95f61a45b1ae15e0742b273f17 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/6d282e95f61a45b1ae15e0742b273f17 2023-04-07 09:56:24,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/e49e1628d4a941e3afa98034955ca944 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/e49e1628d4a941e3afa98034955ca944 2023-04-07 09:56:24,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/09d9cd7b5daf42a781f1214102b93b28 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/09d9cd7b5daf42a781f1214102b93b28 2023-04-07 09:56:24,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/0afb4b815b13464baeb2222d8fb2cbda to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/0afb4b815b13464baeb2222d8fb2cbda 2023-04-07 09:56:24,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a1c96113a1ba45c382b0d5d612c63142 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/a1c96113a1ba45c382b0d5d612c63142 2023-04-07 09:56:24,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/743be0e96ec84edc9debaa03622260c0 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/743be0e96ec84edc9debaa03622260c0 2023-04-07 09:56:24,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8b8559cec5294d7ea354d28a4052491d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/8b8559cec5294d7ea354d28a4052491d 2023-04-07 09:56:24,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/67d45662e46f4e8899d079c64bbb93a3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/67d45662e46f4e8899d079c64bbb93a3 2023-04-07 09:56:24,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/recovered.edits/676.seqid, newMaxSeqId=676, maxSeqId=4 2023-04-07 09:56:24,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df. 2023-04-07 09:56:24,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 448acd9b275d13bcb18a1b50f0aa55df: 2023-04-07 09:56:24,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:24,531 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=448acd9b275d13bcb18a1b50f0aa55df, regionState=CLOSED 2023-04-07 09:56:24,531 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680861384531"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680861384531"}]},"ts":"1680861384531"} 2023-04-07 09:56:24,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-07 09:56:24,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 448acd9b275d13bcb18a1b50f0aa55df, server=jenkins-hbase4.apache.org,35881,1680861218288 in 1.9080 sec 2023-04-07 09:56:24,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-07 09:56:24,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=448acd9b275d13bcb18a1b50f0aa55df, UNASSIGN in 1.9120 sec 2023-04-07 09:56:24,535 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680861384535"}]},"ts":"1680861384535"} 2023-04-07 09:56:24,536 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-07 09:56:24,538 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-07 09:56:24,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9220 sec 2023-04-07 09:56:24,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-07 09:56:24,723 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-07 09:56:24,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-07 09:56:24,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:24,726 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:24,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-07 09:56:24,726 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:24,731 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:24,732 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C, FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/recovered.edits] 2023-04-07 09:56:24,735 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/184af33cb6784de7abc8c2958f8dc241 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/184af33cb6784de7abc8c2958f8dc241 2023-04-07 09:56:24,736 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/555e2241c90346918efa867907e84bdb to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/555e2241c90346918efa867907e84bdb 2023-04-07 09:56:24,737 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/704afeb88bba49e0a7c5e98b237f8ac1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/A/704afeb88bba49e0a7c5e98b237f8ac1 2023-04-07 09:56:24,738 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1f75f9b4d0614625b8f62a17658888f4 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/1f75f9b4d0614625b8f62a17658888f4 2023-04-07 09:56:24,739 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d3a4e70b58c44acebaff8b663c29f08f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/d3a4e70b58c44acebaff8b663c29f08f 2023-04-07 09:56:24,740 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffab3e8c87b141c8bb15666a9b9ea45e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/B/ffab3e8c87b141c8bb15666a9b9ea45e 2023-04-07 09:56:24,741 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1814ac1356ba4983bb11bb9c6baa8890 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/1814ac1356ba4983bb11bb9c6baa8890 2023-04-07 09:56:24,742 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/75b6345a090045499092e2731b5c966a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/75b6345a090045499092e2731b5c966a 2023-04-07 09:56:24,743 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f18f75175821483292ca8eca6f09001a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/C/f18f75175821483292ca8eca6f09001a 2023-04-07 09:56:24,745 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/recovered.edits/676.seqid to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df/recovered.edits/676.seqid 2023-04-07 09:56:24,745 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/.tmp/data/default/TestAcidGuarantees/448acd9b275d13bcb18a1b50f0aa55df 2023-04-07 09:56:24,745 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-07 09:56:24,746 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 09:56:24,746 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-07 09:56:24,748 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040709c5e5d8b57d40989ccc94b1c066a407 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040709c5e5d8b57d40989ccc94b1c066a407 2023-04-07 09:56:24,749 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070f11447944df4a83a02794c5d0e0ff86 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304070f11447944df4a83a02794c5d0e0ff86 2023-04-07 09:56:24,750 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040718fc1202a03d4bbaa03fd3f2c43dfef1 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040718fc1202a03d4bbaa03fd3f2c43dfef1 2023-04-07 09:56:24,751 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407287f76718cb24a6794063e64fef4b68c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407287f76718cb24a6794063e64fef4b68c 2023-04-07 09:56:24,752 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040731f854f4feba4aeebccd9992b95afde2 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040731f854f4feba4aeebccd9992b95afde2 2023-04-07 09:56:24,753 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040736e447325dfb4bc982f5b46ba2e1640f to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040736e447325dfb4bc982f5b46ba2e1640f 2023-04-07 09:56:24,753 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407385675f42e99486fa00067a8fdc6bdb9 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407385675f42e99486fa00067a8fdc6bdb9 2023-04-07 09:56:24,754 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077187bd523427403d979a5d5de46e7f59 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304077187bd523427403d979a5d5de46e7f59 2023-04-07 09:56:24,755 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407864241dc0f2d48a0b15d724f9600cde3 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407864241dc0f2d48a0b15d724f9600cde3 2023-04-07 09:56:24,756 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079be05eba5b0a468f8507f227097efaaf to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304079be05eba5b0a468f8507f227097efaaf 2023-04-07 09:56:24,757 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a312ac9f5567400bb123c07f5a3bfb1e to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a312ac9f5567400bb123c07f5a3bfb1e 2023-04-07 09:56:24,757 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a3fcb05e37cd43738b1bc13cd7c03b85 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a3fcb05e37cd43738b1bc13cd7c03b85 2023-04-07 09:56:24,758 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a9f691a664924e8c91948d518889f79c to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407a9f691a664924e8c91948d518889f79c 2023-04-07 09:56:24,759 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b6bf06f88d244aadb8020b6a7a6d43ec to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407b6bf06f88d244aadb8020b6a7a6d43ec 2023-04-07 09:56:24,760 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d98fc09dceae4181981111fda6e0d1ed to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407d98fc09dceae4181981111fda6e0d1ed 2023-04-07 09:56:24,760 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407da454546d5634daeaeef70f0b40fb8bc to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407da454546d5634daeaeef70f0b40fb8bc 2023-04-07 09:56:24,761 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407dee44c3b3b91440cae74a174679ec291 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407dee44c3b3b91440cae74a174679ec291 2023-04-07 09:56:24,762 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e0312f8c39124354ba8c4667112bd3f7 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e0312f8c39124354ba8c4667112bd3f7 2023-04-07 09:56:24,763 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e449d79b94a64f12b87afbdcbc14df8a to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e449d79b94a64f12b87afbdcbc14df8a 2023-04-07 09:56:24,764 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e60f2bb47c31445d91cb2d4e2b97d68b to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e60f2bb47c31445d91cb2d4e2b97d68b 2023-04-07 09:56:24,765 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e81483028e4c4b32b838d0b21b2b97ab to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407e81483028e4c4b32b838d0b21b2b97ab 2023-04-07 09:56:24,765 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ecccb72dbbbe435e92dda61ac6fd1537 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407ecccb72dbbbe435e92dda61ac6fd1537 2023-04-07 09:56:24,766 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f60cedcf9b49473eaf3accde7e4d3110 to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f60cedcf9b49473eaf3accde7e4d3110 2023-04-07 09:56:24,767 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f6b065379bef45f3adf0b339ec9f66ee to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407f6b065379bef45f3adf0b339ec9f66ee 2023-04-07 09:56:24,768 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fb5a5c61f0794a29a6803f34598ea59d to hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230407fb5a5c61f0794a29a6803f34598ea59d 2023-04-07 09:56:24,768 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-07 09:56:24,770 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:24,771 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-07 09:56:24,772 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-07 09:56:24,773 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:24,773 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-07 09:56:24,773 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680861384773"}]},"ts":"9223372036854775807"} 2023-04-07 09:56:24,774 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-07 09:56:24,774 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 448acd9b275d13bcb18a1b50f0aa55df, NAME => 'TestAcidGuarantees,,1680861358748.448acd9b275d13bcb18a1b50f0aa55df.', STARTKEY => '', ENDKEY => ''}] 2023-04-07 09:56:24,774 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-07 09:56:24,775 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680861384774"}]},"ts":"9223372036854775807"} 2023-04-07 09:56:24,776 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-07 09:56:24,777 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-07 09:56:24,778 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 53 msec 2023-04-07 09:56:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46673] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-07 09:56:24,827 INFO [Listener at localhost/39511] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-07 09:56:24,840 INFO [Listener at localhost/39511] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 228), OpenFileDescriptor=571 (was 587), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=444 (was 404) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=2586 (was 2518) - AvailableMemoryMB LEAK? - 2023-04-07 09:56:24,840 INFO [Listener at localhost/39511] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-07 09:56:24,841 INFO [Listener at localhost/39511] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-07 09:56:24,841 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:49660 2023-04-07 09:56:24,842 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:24,842 DEBUG [Listener at localhost/39511] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-07 09:56:24,842 DEBUG [Listener at localhost/39511] util.JVMClusterUtil(257): Found active master hash=337198405, stopped=false 2023-04-07 09:56:24,842 INFO [Listener at localhost/39511] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:56:24,843 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-07 09:56:24,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 09:56:24,844 INFO [Listener at localhost/39511] procedure2.ProcedureExecutor(629): Stopping 2023-04-07 09:56:24,845 DEBUG [Listener at localhost/39511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08220e1e to 127.0.0.1:49660 2023-04-07 09:56:24,845 DEBUG [Listener at localhost/39511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:24,846 INFO [Listener at localhost/39511] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35881,1680861218288' ***** 2023-04-07 09:56:24,846 INFO [Listener at localhost/39511] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-07 09:56:24,847 INFO [RS:0;jenkins-hbase4:35881] regionserver.HeapMemoryManager(220): Stopping 2023-04-07 09:56:24,847 INFO [RS:0;jenkins-hbase4:35881] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-07 09:56:24,847 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-07 09:56:24,847 INFO [RS:0;jenkins-hbase4:35881] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-07 09:56:24,847 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(3304): Received CLOSE for 18a621e37938d05f473a21997b77c76c 2023-04-07 09:56:24,847 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:24,847 DEBUG [RS:0;jenkins-hbase4:35881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49037e10 to 127.0.0.1:49660 2023-04-07 09:56:24,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 18a621e37938d05f473a21997b77c76c, disabling compactions & flushes 2023-04-07 09:56:24,848 DEBUG [RS:0;jenkins-hbase4:35881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:24,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:56:24,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:56:24,849 INFO [RS:0;jenkins-hbase4:35881] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-07 09:56:24,849 INFO [RS:0;jenkins-hbase4:35881] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-07 09:56:24,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. after waiting 0 ms 2023-04-07 09:56:24,849 INFO [RS:0;jenkins-hbase4:35881] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-07 09:56:24,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:56:24,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 18a621e37938d05f473a21997b77c76c 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-07 09:56:24,849 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-07 09:56:24,850 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-07 09:56:24,850 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 18a621e37938d05f473a21997b77c76c=hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c.} 2023-04-07 09:56:24,851 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-07 09:56:24,851 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-07 09:56:24,851 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-07 09:56:24,851 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-07 09:56:24,851 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-07 09:56:24,851 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-07 09:56:24,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-07 09:56:24,852 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:56:24,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-07 09:56:24,852 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1505): Waiting on 1588230740, 18a621e37938d05f473a21997b77c76c 2023-04-07 09:56:24,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/.tmp/info/0be5beec0348400f91aa9d9159fc6523 2023-04-07 09:56:24,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/.tmp/info/0be5beec0348400f91aa9d9159fc6523 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/info/0be5beec0348400f91aa9d9159fc6523 2023-04-07 09:56:24,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/info/0be5beec0348400f91aa9d9159fc6523, entries=2, sequenceid=6, filesize=4.8 K 2023-04-07 09:56:24,882 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 18a621e37938d05f473a21997b77c76c in 33ms, sequenceid=6, compaction requested=false 2023-04-07 09:56:24,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/namespace/18a621e37938d05f473a21997b77c76c/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-07 09:56:24,890 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:56:24,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 18a621e37938d05f473a21997b77c76c: 2023-04-07 09:56:24,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680861221696.18a621e37938d05f473a21997b77c76c. 2023-04-07 09:56:24,943 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-07 09:56:25,053 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 09:56:25,253 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 09:56:25,266 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/.tmp/info/166c7e87c4c144a580109f326ce6b126 2023-04-07 09:56:25,269 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 166c7e87c4c144a580109f326ce6b126 2023-04-07 09:56:25,430 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-07 09:56:25,430 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-07 09:56:25,453 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 09:56:25,654 DEBUG [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-07 09:56:25,674 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/.tmp/rep_barrier/dd038cd0736d4d73ba94359ea5b03ece 2023-04-07 09:56:25,677 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for dd038cd0736d4d73ba94359ea5b03ece 2023-04-07 09:56:25,682 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/.tmp/table/0037e156b36c4481b624e54472265aff 2023-04-07 09:56:25,685 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0037e156b36c4481b624e54472265aff 2023-04-07 09:56:25,685 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/.tmp/info/166c7e87c4c144a580109f326ce6b126 as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/info/166c7e87c4c144a580109f326ce6b126 2023-04-07 09:56:25,691 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 166c7e87c4c144a580109f326ce6b126 2023-04-07 09:56:25,691 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/info/166c7e87c4c144a580109f326ce6b126, entries=22, sequenceid=93, filesize=7.1 K 2023-04-07 09:56:25,694 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/.tmp/rep_barrier/dd038cd0736d4d73ba94359ea5b03ece as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/rep_barrier/dd038cd0736d4d73ba94359ea5b03ece 2023-04-07 09:56:25,697 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for dd038cd0736d4d73ba94359ea5b03ece 2023-04-07 09:56:25,697 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/rep_barrier/dd038cd0736d4d73ba94359ea5b03ece, entries=6, sequenceid=93, filesize=5.5 K 2023-04-07 09:56:25,698 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/.tmp/table/0037e156b36c4481b624e54472265aff as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/table/0037e156b36c4481b624e54472265aff 2023-04-07 09:56:25,701 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0037e156b36c4481b624e54472265aff 2023-04-07 09:56:25,701 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/table/0037e156b36c4481b624e54472265aff, entries=9, sequenceid=93, filesize=5.5 K 2023-04-07 09:56:25,702 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 851ms, sequenceid=93, compaction requested=false 2023-04-07 09:56:25,716 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-07 09:56:25,717 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-07 09:56:25,718 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-07 09:56:25,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-07 09:56:25,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-07 09:56:25,854 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35881,1680861218288; all regions closed. 2023-04-07 09:56:25,862 DEBUG [RS:0;jenkins-hbase4:35881] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/oldWALs 2023-04-07 09:56:25,862 INFO [RS:0;jenkins-hbase4:35881] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35881%2C1680861218288.meta:.meta(num 1680861221442) 2023-04-07 09:56:25,867 DEBUG [RS:0;jenkins-hbase4:35881] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/oldWALs 2023-04-07 09:56:25,868 INFO [RS:0;jenkins-hbase4:35881] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35881%2C1680861218288:(num 1680861220527) 2023-04-07 09:56:25,868 DEBUG [RS:0;jenkins-hbase4:35881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:25,868 INFO [RS:0;jenkins-hbase4:35881] regionserver.LeaseManager(133): Closed leases 2023-04-07 09:56:25,868 INFO [RS:0;jenkins-hbase4:35881] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-07 09:56:25,868 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-07 09:56:25,870 INFO [RS:0;jenkins-hbase4:35881] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35881 2023-04-07 09:56:25,876 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-07 09:56:25,876 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35881,1680861218288 2023-04-07 09:56:25,876 ERROR [Listener at localhost/39511-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@59a6e3d5 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2972] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 09:56:25,877 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-07 09:56:25,877 ERROR [Listener at localhost/39511-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@471f7eeb rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2972] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 09:56:25,878 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35881,1680861218288] 2023-04-07 09:56:25,878 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35881,1680861218288; numProcessing=1 2023-04-07 09:56:25,880 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35881,1680861218288 already deleted, retry=false 2023-04-07 09:56:25,880 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35881,1680861218288 expired; onlineServers=0 2023-04-07 09:56:25,880 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46673,1680861217027' ***** 2023-04-07 09:56:25,880 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-07 09:56:25,880 DEBUG [M:0;jenkins-hbase4:46673] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5524377e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-07 09:56:25,880 INFO [M:0;jenkins-hbase4:46673] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46673,1680861217027 2023-04-07 09:56:25,880 INFO [M:0;jenkins-hbase4:46673] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46673,1680861217027; all regions closed. 2023-04-07 09:56:25,880 DEBUG [M:0;jenkins-hbase4:46673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-07 09:56:25,880 DEBUG [M:0;jenkins-hbase4:46673] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-07 09:56:25,880 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-07 09:56:25,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680861220179] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680861220179,5,FailOnTimeoutGroup] 2023-04-07 09:56:25,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680861220180] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680861220180,5,FailOnTimeoutGroup] 2023-04-07 09:56:25,880 DEBUG [M:0;jenkins-hbase4:46673] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-07 09:56:25,881 INFO [M:0;jenkins-hbase4:46673] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-07 09:56:25,881 INFO [M:0;jenkins-hbase4:46673] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-07 09:56:25,881 INFO [M:0;jenkins-hbase4:46673] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-07 09:56:25,881 DEBUG [M:0;jenkins-hbase4:46673] master.HMaster(1512): Stopping service threads 2023-04-07 09:56:25,881 INFO [M:0;jenkins-hbase4:46673] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-07 09:56:25,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-07 09:56:25,881 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-07 09:56:25,882 ERROR [M:0;jenkins-hbase4:46673] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (249600520) connection to localhost/127.0.0.1:36015 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-07 09:56:25,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-07 09:56:25,882 INFO [M:0;jenkins-hbase4:46673] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-07 09:56:25,882 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-07 09:56:25,882 DEBUG [M:0;jenkins-hbase4:46673] zookeeper.ZKUtil(398): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-07 09:56:25,883 WARN [M:0;jenkins-hbase4:46673] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-07 09:56:25,883 INFO [M:0;jenkins-hbase4:46673] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-07 09:56:25,883 INFO [M:0;jenkins-hbase4:46673] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-07 09:56:25,883 DEBUG [M:0;jenkins-hbase4:46673] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-07 09:56:25,883 INFO [M:0;jenkins-hbase4:46673] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:56:25,883 DEBUG [M:0;jenkins-hbase4:46673] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:56:25,883 DEBUG [M:0;jenkins-hbase4:46673] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-07 09:56:25,883 DEBUG [M:0;jenkins-hbase4:46673] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:56:25,883 INFO [M:0;jenkins-hbase4:46673] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.57 KB heapSize=393.29 KB 2023-04-07 09:56:25,898 INFO [M:0;jenkins-hbase4:46673] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.57 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f79464f3129940219e1dd458ae39257c 2023-04-07 09:56:25,901 INFO [M:0;jenkins-hbase4:46673] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f79464f3129940219e1dd458ae39257c 2023-04-07 09:56:25,902 DEBUG [M:0;jenkins-hbase4:46673] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/f79464f3129940219e1dd458ae39257c as hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f79464f3129940219e1dd458ae39257c 2023-04-07 09:56:25,905 INFO [M:0;jenkins-hbase4:46673] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f79464f3129940219e1dd458ae39257c 2023-04-07 09:56:25,905 INFO [M:0;jenkins-hbase4:46673] regionserver.HStore(1080): Added hdfs://localhost:36015/user/jenkins/test-data/0ae5c031-e1f2-136c-eed5-859afba95f2e/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/f79464f3129940219e1dd458ae39257c, entries=65, sequenceid=578, filesize=24.6 K 2023-04-07 09:56:25,906 INFO [M:0;jenkins-hbase4:46673] regionserver.HRegion(2948): Finished flush of dataSize ~344.57 KB/352840, heapSize ~393.27 KB/402712, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 23ms, sequenceid=578, compaction requested=false 2023-04-07 09:56:25,907 INFO [M:0;jenkins-hbase4:46673] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-07 09:56:25,907 DEBUG [M:0;jenkins-hbase4:46673] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-07 09:56:25,910 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-07 09:56:25,910 INFO [M:0;jenkins-hbase4:46673] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-07 09:56:25,911 INFO [M:0;jenkins-hbase4:46673] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46673 2023-04-07 09:56:25,913 DEBUG [M:0;jenkins-hbase4:46673] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46673,1680861217027 already deleted, retry=false 2023-04-07 09:56:25,979 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 09:56:25,979 INFO [RS:0;jenkins-hbase4:35881] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35881,1680861218288; zookeeper connection closed. 2023-04-07 09:56:25,979 ERROR [Listener at localhost/39511-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f275c32 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2972] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 09:56:25,980 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): regionserver:35881-0x1015e0aa7780001, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 09:56:25,980 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@fb09b3a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@fb09b3a 2023-04-07 09:56:25,980 ERROR [Listener at localhost/39511-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4460d77a rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2972] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 09:56:25,980 INFO [Listener at localhost/39511] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-07 09:56:26,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 09:56:26,079 INFO [M:0;jenkins-hbase4:46673] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46673,1680861217027; zookeeper connection closed. 2023-04-07 09:56:26,079 ERROR [Listener at localhost/39511-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ab70e0d rejected from java.util.concurrent.ThreadPoolExecutor@54f15a6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4960] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 09:56:26,079 DEBUG [Listener at localhost/39511-EventThread] zookeeper.ZKWatcher(600): master:46673-0x1015e0aa7780000, quorum=127.0.0.1:49660, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-07 09:56:26,080 ERROR [Listener at localhost/39511-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@538c6d89 rejected from java.util.concurrent.ThreadPoolExecutor@54f15a6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4960] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-07 09:56:26,081 WARN [Listener at localhost/39511] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-07 09:56:26,084 INFO [Listener at localhost/39511] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-07 09:56:26,193 WARN [BP-1602883843-172.31.14.131-1680861213773 heartbeating to localhost/127.0.0.1:36015] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-07 09:56:26,193 WARN [BP-1602883843-172.31.14.131-1680861213773 heartbeating to localhost/127.0.0.1:36015] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1602883843-172.31.14.131-1680861213773 (Datanode Uuid 70ccd2b6-8a23-49ed-96bf-90cb8121a8b4) service to localhost/127.0.0.1:36015 2023-04-07 09:56:26,196 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/cluster_33ba0ff8-c2ef-b3a6-4e4d-4cc1067ceb2e/dfs/data/data1/current/BP-1602883843-172.31.14.131-1680861213773] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-07 09:56:26,197 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/acceaadc-7006-82c8-831b-0cd8269689b1/cluster_33ba0ff8-c2ef-b3a6-4e4d-4cc1067ceb2e/dfs/data/data2/current/BP-1602883843-172.31.14.131-1680861213773] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-07 09:56:26,240 INFO [Listener at localhost/39511] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-07 09:56:26,420 INFO [Listener at localhost/39511] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-07 09:56:26,622 INFO [Listener at localhost/39511] hbase.HBaseTestingUtility(1293): Minicluster is down